Projects:2015s1-31 Cracking the Voynich manuscript code - Revision history

A1211832: /* Approach */

2015-10-23T01:42:54Z

‎Approach

A1211832: /* Introduction */

2015-10-23T01:40:48Z

‎Introduction

A1211832: /* Introduction */

2015-10-23T01:40:33Z

‎Introduction

A1211832: /* Introduction */

2015-10-23T01:39:01Z

‎Introduction

A1211832: /* Background */

2015-10-23T01:37:07Z

‎Background

A1211832: /* References */

2015-10-23T01:34:49Z

‎References

A1211832: /* References */

2015-10-23T01:34:35Z

‎References

A1211832 at 01:33, 23 October 2015

2015-10-23T01:33:52Z

A1211832: /* References */

2015-10-23T01:27:06Z

‎References

A1211832: /* Phase 5: Collocations */

2015-10-23T01:22:08Z

‎Phase 5: Collocations

← Older revision		Revision as of 01:42, 23 October 2015
Line 159:		Line 159:

	Where P(x,y) is the probability of two words appearing coincidentally int the text and P(x) and P(y) are the probabilities of each word appearing within the text.		Where P(x,y) is the probability of two words appearing coincidentally int the text and P(x) and P(y) are the probabilities of each word appearing within the text.
		+
		+	==Results==
		+
		+	For details and discussions of the results please visit [https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/Cracking_the_Voynich_Code_2015_-_Final_Report our report]. Concluding statements of these results can be found below.

	==Deliverables==		==Deliverables==

@@ Line 4: / Line 4: @@
 An unknown script that has been a mystery for several decades, powerful computational tools and access to a large array of data on the internet. This project looks at the famous linguistic and cryptologic mystery, The Voynich Manuscript, and attempts to find possible relationships to other languages from a basic statistical perspective. Utilizing a digital transcription of the Voynich, statistics are data-mined and analysed in various techniques and compared with any previous research available on the internet and a corpus of texts from various languages.
-[[File:Astronomical Section.jpg|500px|thumb|left|An Example (Astronomical) Section of the Voynich Manuscript]]
+[[File:Astronomical Section.jpg|500px|thumb|center|An Example (Astronomical) Section of the Voynich Manuscript]]
 =Team=

@@ Line 4: / Line 4: @@
 An unknown script that has been a mystery for several decades, powerful computational tools and access to a large array of data on the internet. This project looks at the famous linguistic and cryptologic mystery, The Voynich Manuscript, and attempts to find possible relationships to other languages from a basic statistical perspective. Utilizing a digital transcription of the Voynich, statistics are data-mined and analysed in various techniques and compared with any previous research available on the internet and a corpus of texts from various languages.
-[[File:Astronomical Section.jpg]]
+[[File:Astronomical Section.jpg|500px|thumb|left|An Example (Astronomical) Section of the Voynich Manuscript]]
 =Team=

@@ Line 3: / Line 3: @@
 =Introduction=
 An unknown script that has been a mystery for several decades, powerful computational tools and access to a large array of data on the internet. This project looks at the famous linguistic and cryptologic mystery, The Voynich Manuscript, and attempts to find possible relationships to other languages from a basic statistical perspective. Utilizing a digital transcription of the Voynich, statistics are data-mined and analysed in various techniques and compared with any previous research available on the internet and a corpus of texts from various languages.
 =Team=

@@ Line 22: / Line 22: @@
 The Voynich Manuscript is a document written in an unknown script that has been carbon dated back to the early 15th century <ref name=Stolte/> and believed to be created within Europe <ref name=Reddy/>. Named after Wilfrid Voynich, whom purchased the folio in 1912, the manuscript has become a well-known mystery within linguistics and cryptology. It is divided into several different sections based on the nature of the drawings <ref name=Landini/>. These sections are:
-*Herbal
+*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/File:Herbal_Section.jpg Herbal]
-*Astronomical
+*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/File:Astronomical_Section.jpg Astronomical]
-*Biological
+*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/File:Biological_Section.jpg Biological]
-*Cosmological
+*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/File:Cosmological_Section.jpg Cosmological]
-*Pharmaceutical
+*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/File:Pharmaceutical_Section.jpg Pharmaceutical]
-*Recipes
+*[https://www.eleceng.adelaide.edu.au/personal/dabbott/wiki/index.php/File:Recipes_Section.jpg Recipes]
 The folio numbers and examples of each section are outlined in appendix section A.2. In general, the Voynich Manuscript has fallen into three particular hypotheses <ref name=Schinner/>. These are as follows:

← Older revision		Revision as of 01:34, 23 October 2015
Line 255:		Line 255:

	<ref name=Data> S. Chakrabarti, M. Ester, U. Fayyad, J. Gehrke, J. Han, S. Morishita, G. Piatetsky-Shapiro and W. Wang, “Data Mining Curriculum: A Proposal (Version 1.0),” 12 April 2015. [Online]. Available: http://www.kdd.org/curriculum/index.html.</ref>		<ref name=Data> S. Chakrabarti, M. Ester, U. Fayyad, J. Gehrke, J. Han, S. Morishita, G. Piatetsky-Shapiro and W. Wang, “Data Mining Curriculum: A Proposal (Version 1.0),” 12 April 2015. [Online]. Available: http://www.kdd.org/curriculum/index.html.</ref>
−
−	~~<ref name=Goldsmith> J. Goldsmith, “Unsupervised Learning of the Morphology of a Natural Language,” Computational Linguistics, pp. 153-198, 2001. </ref>~~

	<ref name=Currier> P. Currier, “New Research on the Voynich Manuscript: Proceedings of a Seminar,” 30 November 1976. [Online]. Available: http://www.voynich.nu/extra/curr_main.html. </ref>		<ref name=Currier> P. Currier, “New Research on the Voynich Manuscript: Proceedings of a Seminar,” 30 November 1976. [Online]. Available: http://www.voynich.nu/extra/curr_main.html. </ref>

@@ Line 256: / Line 256: @@
 <ref name=Data> S. Chakrabarti, M. Ester, U. Fayyad, J. Gehrke, J. Han, S. Morishita, G. Piatetsky-Shapiro and W. Wang, “Data Mining Curriculum: A Proposal (Version 1.0),” 12 April 2015. [Online]. Available: http://www.kdd.org/curriculum/index.html.</ref>
-<ref name=Golsmith> J. Goldsmith, “Unsupervised Learning of the Morphology of a Natural Language,” Computational Linguistics, pp. 153-198, 2001. </ref>
+<ref name=Goldsmith> J. Goldsmith, “Unsupervised Learning of the Morphology of a Natural Language,” Computational Linguistics, pp. 153-198, 2001. </ref>
 <ref name=Currier> P. Currier, “New Research on the Voynich Manuscript: Proceedings of a Seminar,” 30 November 1976. [Online]. Available: http://www.voynich.nu/extra/curr_main.html. </ref>

@@ Line 21: / Line 21: @@
 ==Background==
-The Voynich Manuscript is a document written in an unknown script that has been carbon dated back to the early 15th century [1] and believed to be created within Europe [2]. Named after Wilfrid Voynich, whom purchased the folio in 1912, the manuscript has become a well-known mystery within linguistics and cryptology. It is divided into several different sections based on the nature of the drawings [3]. These sections are:
+The Voynich Manuscript is a document written in an unknown script that has been carbon dated back to the early 15th century <ref name=Stolte/> and believed to be created within Europe <ref name=Reddy/>. Named after Wilfrid Voynich, whom purchased the folio in 1912, the manuscript has become a well-known mystery within linguistics and cryptology. It is divided into several different sections based on the nature of the drawings <ref name=Landini/>. These sections are:
 *Herbal
 *Astronomical
@@ Line 29: / Line 29: @@
 *Recipes
-The folio numbers and examples of each section are outlined in appendix section A.2. In general, the Voynich Manuscript has fallen into three particular hypotheses [4]. These are as follows:
+The folio numbers and examples of each section are outlined in appendix section A.2. In general, the Voynich Manuscript has fallen into three particular hypotheses <ref name=Schinner/>. These are as follows:
 *Cipher Text: The text is encrypted.
 *Plain Text: The text is in a plain, natural language that is currently unidentified.
 *Hoax: The text has no meaningful information.
-Note that the manuscript may fall into more than one of these hypotheses [4]. It may be that the manuscript is written through steganography, the concealing of the true meaning within the possibly meaningless text.
+Note that the manuscript may fall into more than one of these hypotheses <ref name=Schinner/>. It may be that the manuscript is written through steganography, the concealing of the true meaning within the possibly meaningless text.
 ==Technical Background==
-The vast majority of the project relies on a technique known as data mining. Data mining is the process of taking and analysing a large data set in order to uncover particular patterns and correlations within said data thus creating useful knowledge [6]. In terms of the project, data shall be acquired from the Interlinear Archive, a digital archive of transcriptions from the Voynich Manuscript, and other sources of digital texts in known languages. Data mined from the Interlinear Archive will be tested and analysed for specific linguistic properties using varying statistical methods.
+The vast majority of the project relies on a technique known as data mining. Data mining is the process of taking and analysing a large data set in order to uncover particular patterns and correlations within said data thus creating useful knowledge <ref name=Data/>. In terms of the project, data shall be acquired from the Interlinear Archive, a digital archive of transcriptions from the Voynich Manuscript, and other sources of digital texts in known languages. Data mined from the Interlinear Archive will be tested and analysed for specific linguistic properties using varying statistical methods.
 The Interlinear Archive, as mentioned, will be the main source of data in regards to the Voynich Manuscript. It has been compiled to be a machine readable version of the Voynich Manuscript based on transcriptions from various transcribers. Each transcription has been translated into the [http://www.voynich.nu/extra/eva.html European Voynich Alphabet (EVA)].
@@ Line 51: / Line 51: @@
 The project attempts to find relationships and patterns within unknown text through the usage of basic linguistic properties and analyses. The Voynich Manuscript is a prime candidate for analyses as there is no known accepted translations of any part within the document. The relationships found can be used help narrow future research and to conclude on specific features of the unknown language within the Voynich Manuscript.
-Knowledge produced from the relationships and patterns of languages and linguistics can be used to further the current linguistic computation and encryption/decryption technologies of today [5].
+Knowledge produced from the relationships and patterns of languages and linguistics can be used to further the current linguistic computation and encryption/decryption technologies of today <ref name=Amancio/>.
 While some may question as to why an unknown text is of any importance to Engineering, a more general view of the research project shows that it deals with data acquisition and analyses. This is integral to a wide array of businesses, including engineering, which can involve a basic service, such as survey analysis, to more complex automated system.
@@ Line 57: / Line 57: @@
 ==Significance==
-There are many computational linguistic and encryption/decryption technologies that are in use today. As mentioned in section 1.3, knowledge produced from this research can help advance these technologies in a range of different applications [5]. These include, but are not limited to, information retrieval systems, search engines, machine translators, automatic summarizers, and social networks [5].
+There are many computational linguistic and encryption/decryption technologies that are in use today. As mentioned in section 1.3, knowledge produced from this research can help advance these technologies in a range of different applications <ref name=Amancio/>. These include, but are not limited to, information retrieval systems, search engines, machine translators, automatic summarizers, and social networks <ref name=Amancio/>.
 Particular technologies, that are widely used today, that can benefit from the research, include:
@@ Line 126: / Line 126: @@
 ===Phase 3 - Morphology Investigation===
-Linguistic morphology, broadly speaking, deals with the study of the internal structure of the words, particularly the meaningful segments that make up a word [17]. Specifically, phase 3 will be looking into the possibility of affixes within the Voynich Manuscript from a strictly concatenative perspective.
+Linguistic morphology, broadly speaking, deals with the study of the internal structure of the words, particularly the meaningful segments that make up a word <ref name=Aronoff/>. Specifically, phase 3 will be looking into the possibility of affixes within the Voynich Manuscript from a strictly concatenative perspective.
-Previous research has found the possibility of morphological structure within the Voynich Manuscript [2]. Within this small experiment, the most common affixes in English are found and compared with those found within the Voynich Manuscript. Due to the unknown word structure and small relative size of the Voynich, this experiment defines an affix as a sequence of characters that appear at the word edges. Using this basic affix definition, a simple ranking of the affixes of various lengths could reveal potential relationships between the Voynich and other known languages.
+Previous research has found the possibility of morphological structure within the Voynich Manuscript <ref name=Reddy/>. Within this small experiment, the most common affixes in English are found and compared with those found within the Voynich Manuscript. Due to the unknown word structure and small relative size of the Voynich, this experiment defines an affix as a sequence of characters that appear at the word edges. Using this basic affix definition, a simple ranking of the affixes of various lengths could reveal potential relationships between the Voynich and other known languages.
 The basis of the code extracts and counts each possible character sequence within a word for a specific character sequence length. These are then ranked, according to frequency, and compared with all languages tested. By comparing the relative frequencies and their difference ratios, we are able to find if any of the tested languages share a possible relationship with the Voynich Manuscript.
@@ Line 145: / Line 145: @@
 *Determine the frequency of a given word token
-The resulting statistics from the code can then be used into investigation. However, it should be noted that the manuscript may have been written by multiple authors and in multiple languages [13]. Sections of the manuscript will need to be investigated separately, particularly those written in different languages, along with the manuscript as a whole.
+The resulting statistics from the code can then be used into investigation. However, it should be noted that the manuscript may have been written by multiple authors and in multiple languages <ref name=Currier/>. Sections of the manuscript will need to be investigated separately, particularly those written in different languages, along with the manuscript as a whole.
 ===Phase 5: Collocations===
-Collocations have no universally accepted formal definition but deals with the words within a language that co-occur more often than would be expected by chance. Natural languages are full of collocations and can vary significantly depending on the metric, such as length or pattern, used to define a collocation.
+Collocations have no universally accepted formal definition <ref name=Melcuk/> but deals with the words within a language that co-occur more often than would be expected by chance <ref name=Smadja/>. Natural languages are full of collocations <ref name=Smadja/> and can vary significantly depending on the metric, such as length or pattern, used to define a collocation <ref name=ThanopoulosFakotakisKokkinakis/>.
 In this research experiment, the definition used for a collocation is that of two words occurring directly next to each other. As collocations have varying significance within different languages, by extracting and comparing all possible collocations within the Voynich Manuscript and the corpus, a relationship based on word association could be found or provide evidence of the possibility of a hoax.
-Initially collocations are ranked by frequency, from most frequent to least frequent, followed by a Pointwise Mutual Information (PMI) metric. PMI is a widely accepted method to quantify the strength of word association and is mathematically defined as:
+Initially collocations are ranked by frequency, from most frequent to least frequent, followed by a Pointwise Mutual Information (PMI) metric. PMI is a widely accepted method to quantify the strength of word association <ref name=ThanopoulosFakotakisKokkinakis/> <ref name=WermterHahn/> and is mathematically defined as:
 [[File:PMI.png]]
@@ Line 261: / Line 261: @@
 <ref name=Aronoff> M. Aronoff, and K. Fudeman, “What is morphology,” Vol. 8., John Wiley & Sons, pp. 1-25, 2011.</ref>
 }}

@@ Line 242: / Line 242: @@
 =References=
-[1] D. Stolte, “Experts determine age of book 'nobody can read',” 10 February 2011. [Online]. Available: http://phys.org/news/2011-02-experts-age.html. [Accessed 12 March 2015].
+{{Reflist|colwidth=30em|refs=
-[2] S. Reddy and K. Knight, “What We Know About The Voynich Manuscript,” LaTeCH '11 Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pp. 78-86, 2011.
+<ref name=Stolte> D. Stolte, “Experts determine age of book 'nobody can read',” 10 February 2011. [Online]. Available: http://phys.org/news/2011-02-experts-age.html. [Accessed 12 March 2015]. </ref>
-[3] G. Landini, “Evidence Of Linguistic Structure In The Voynich Manuscript Using Spectral Analysis,” Cryptologia, pp. 275-295, 2001.
+<ref name=Reddy> S. Reddy and K. Knight, “What We Know About The Voynich Manuscript,” LaTeCH '11 Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pp. 78-86, 2011. </ref>
-[4] A. Schinner, “The Voynich Manuscript: Evidence of the Hoax Hypothesis,” Cryptologia, pp. 95-107, 2007.
+<ref name=Landini> G. Landini, “Evidence Of Linguistic Structure In The Voynich Manuscript Using Spectral Analysis,” Cryptologia, pp. 275-295, 2001. </ref>
-[5] D. R. Amancio, E. G. Altmann, D. Rybski, O. N. Oliveira Jr. and L. d. F. Costa, “Probing the Statistical Properties of Unknown Texts: Application to the Voynich Manuscript,” PLoS ONE 8(7), vol. 8, no. 7, pp. 1-10, 2013.
+<ref name=Schinner> A. Schinner, “The Voynich Manuscript: Evidence of the Hoax Hypothesis,” Cryptologia, pp. 95-107, 2007.</ref>
-[6] S. Chakrabarti, M. Ester, U. Fayyad, J. Gehrke, J. Han, S. Morishita, G. Piatetsky-Shapiro and W. Wang, “Data Mining Curriculum: A Proposal (Version 1.0),” 12 April 2015. [Online]. Available: http://www.kdd.org/curriculum/index.html.
+<ref name=Amancio> D. R. Amancio, E. G. Altmann, D. Rybski, O. N. Oliveira Jr. and L. d. F. Costa, “Probing the Statistical Properties of Unknown Texts: Application to the Voynich Manuscript,” PLoS ONE 8(7), vol. 8, no. 7, pp. 1-10, 2013. </ref>
-[12] J. Goldsmith, “Unsupervised Learning of the Morphology of a Natural Language,” Computational Linguistics, pp. 153-198, 2001.
+<ref name=Data> S. Chakrabarti, M. Ester, U. Fayyad, J. Gehrke, J. Han, S. Morishita, G. Piatetsky-Shapiro and W. Wang, “Data Mining Curriculum: A Proposal (Version 1.0),” 12 April 2015. [Online]. Available: http://www.kdd.org/curriculum/index.html.</ref>
-[13] P. Currier, “New Research on the Voynich Manuscript: Proceedings of a Seminar,” 30 November 1976. [Online]. Available: http://www.voynich.nu/extra/curr_main.html.
+<ref name=Golsmith> J. Goldsmith, “Unsupervised Learning of the Morphology of a Natural Language,” Computational Linguistics, pp. 153-198, 2001. </ref>
-[17] M. Aronoff, and K. Fudeman, “What is morphology,” Vol. 8., John Wiley & Sons, pp. 1-25, 2011.
+<ref name=Currier> P. Currier, “New Research on the Voynich Manuscript: Proceedings of a Seminar,” 30 November 1976. [Online]. Available: http://www.voynich.nu/extra/curr_main.html. </ref>
 =Back=
 *[https://www.eleceng.adelaide.edu.au/students/wiki/projects/index.php/Main_Page Projects Main Page]

@@ Line 154: / Line 154: @@
 Initially collocations are ranked by frequency, from most frequent to least frequent, followed by a Pointwise Mutual Information (PMI) metric. PMI is a widely accepted method to quantify the strength of word association and is mathematically defined as:
-[[File:PMI.jpg]]
+[[File:PMI.png]]
 Where P(x,y) is the probability of two words appearing coincidentally int the text and P(x) and P(y) are the probabilities of each word appearing within the text.