Projects:2014S1-44 Cracking the Voynich Manuscript Code
The Voynich Manuscript is a mysterious 15th century manuscript written in an unknown alphabet. The mysterious nature of the manuscript has attracted linguists and cryptographers over the course of the last hundred years, each attempting to "Crack the Voynich code". However, despite a great deal of research into the text and illustrations of the manuscript, no one has ever conclusively deciphered a single word.
Fortunately the whole book has been converted into an electronic format with each character changed to a convenient ascii character.
This project expands upon past research into the linguistic features of the manuscript. Computational analysis techniques such as Word Recurrence Intervals and N-gram relationships, along with supervised learning algorithms such as Support Vector Machines (SVM) and Multiple Discriminant Analysis (MDA) are applied to an electronic transcription of the text. The team evaluates the use of these classification methods, and also develops new ways to identify grammar and syntax in the Voynich language.
Contents
Project information
Background
The Voynich Manuscript is a mysterious book written in an unknown alphabet. So little is known about the nature and origins of the manuscript that it has been named after Wifred Michael Voynich, the collector who purchased it in 1912 from a castle in Italy. The manuscript has been verified by radiocarbon dating (at the University of Arizona) as belonging to the early 15th century[1], and appears to be a herbal (or medicinal) manual from this time period in Europe. However, despite a great deal of research into the text and illustrations of the manuscript, no one has ever conclusively deciphered a single word. The mysterious nature of the manuscript has attracted linguists and cryptographers over the course of the last hundred years, each attempting to "crack the Voynich code".
The manuscript itself is made up of several folios, numbered from f1 to f116. Each folio consists of two pages (labelled r and v), with the exception of ten foldouts of up to six pages. Although the page numbering was clearly added after the manuscript was written, there are gaps in the numbering which indicate missing folios, and indications that some folios have been reordered long after their completion. These oddities have led scholars to believe that the manuscript may have had several owners. Certain pages also contain a "key-like" sequence of characters, which is why many leading cryptographers of the time (and indeed, today) believe it to be a key cipher. [2]
Between 1996 and 1998, several academics began work on what they hoped to be a complete database of Voynich transcriptions. This database was known as the “Interlinear Archive of Electronic Transcriptions in EVA” (referred to as the Interlinear Archive), and came to include several partial transcriptions such as those developed by Currier and Friedman, along with a complete transcription by Takeshi Takahashi. [3] All the transcriptions in this file were converted to a newly developed alphabet, called EVA, that attempted to correct the errors of previous alphabets (which often oversimplified the characters in the manuscript and ignored rare characters) by using ligature analysis on the handwriting. [4]
The author of the manuscript has been heavily disputed. Some popular candidates are Roger Bacon, John Dee, and Leonardo da Vinci.[5]
Technical Background
This project is heavily dependent on data mining techniques, as it involves the analysis and extraction of large quantities of data from different sources. Data mining can be used to analyse and predict behaviours, find hidden patterns and meanings or compare different data sets for commonalities or correlation. Authorship detection (stylometry) involves a subset of data mining techniques that help determine the authenticity of works and the possible authors of undocumented texts. The techniques involved in this project are also used in applications such as search engine development, code analysis, language processing, and plagiarism detection.
Part of the complexity involved in the VMS decoding process is the difficulty of transcription. The VMS has been transcribed many times by many different scholars, each attempting to create a universally acceptable transcription, but each of these has had to make limiting decisions about the structure of the manuscript. Due to the nature of the writing, the two characters observed by one scholar may also be interpreted as one combined character, and the spacing between words in the manuscript is notoriously vague. This ambiguity reduces the effectiveness of standard cryptographic and linguistic techniques to observe the relationship between characters and between separate word tokens.
Project Objectives
As this is the first year in which an Honours Project has investigated the Voynich Manuscript, the long term objectives have been flexible and subject to change. Furthermore, given the large body of work already in existence on the manuscript, it was unlikely that even a partial decoding could be produced by our team within a year. Instead the primary focus of this project was to research and understand some features of the manuscript, and effectively analyse our data. We aimed to develop ideas and code which could contribute to the overall project outcome and aid future students with their research. The broad goals of our project included:
- Developing possible methods and statistics that could be used to compare an unknown language with a known language or data set
- Comparing the linguistic characteristics and features of the Voynich Manuscript with relevant languages and authors
- Theorising as to whether the language contained within the manuscript is real, a code or a hoax
- Developing a code base, documentation, and clear analysis to aid future projects
That being said, we had certain concrete milestones that we hoped to achieve this year, and certain research areas which we considered to be most worthwhile. These included:
- Word Recurrence Interval as a language independent statistic
- Identification of words which may relate to the illustrations in the manuscript
- SVM and MDA text classification
Approach and Stages
In the early planning stages, we looked at our goals (concrete and flexible) and decided to divide our work into five stages. Each stage was dependent on areas from the previous stages and allowed continuous developments and improvements as our understanding of the manuscript progressed. Early stages involved basic characterisation of the manuscript, and later stages involved text classifications algorithms and research-based strategies. The stages themselves were split evenly between team members (detailed in the section of Project Management) and was designed to take advantage of the unique skills of each team member. The five stages are listed below:
- Phase 1: Characterize the text. Write scripts that count its features. How many words? How long is the alphabet? Word frequencies? Probability of one letter following another. Probability of two letter pairs (2-grams) and n-letter group (n-grams). Compare these in a table with known languages obtained by running your same code on the Declaration of Human Rights. Don't forget to get a short paragraph of English and manually count everything and then run it on your code to cross check it is counting correctly. You must always validate your code or you will lose marks.
- Phase 2: Write a general descriptor for each picture in the book, eg. water, woman, tree, flower, vegetable, leaf, dancing etc. Associate each descriptor with the appropriate paper. Write some code to find which words on a page are unique to those pages with those descriptors. Which words also suddenly increase in frequency on those pages with shared descriptors? Tabulate the results.
- Phase 3: Investigate the use of Word Recurrence Interval (WRI) versus rank plots. Plot WRI curves of the Voynich versus other languages from the Declaration of Human Rights.
- Phase 4: Think up some other ideas to try out.
- Phase 5: As WRI is a language-independent metric, you can select classification features based on WRI. Then you can run an SVM and an MDA classifier to compare the Voynich against other languages in the Declaration of Human Rights. Then you can run it against the works of specific authors of interest such as Roger Bacon, John Dee, and Edward Kelley.
Deliverables and Progress
- Proposal seminar
- Progress report
- Final seminar
- Final report
- Poster
- Project exhibition 'expo'
- YouTube video
- Weekly progress
Follow this link for current progress
End results
It will familiarize you with techniques in information theory, probability, statistics, encryption, decryption, signal classification, and datamining. It will also improve your software skills. The new software tools you develop may lead to new IP in the areas of datamining, automatic text language identification, and also make you rich/famous. The types of jobs out there where these skills are useful are in computer security, comms, digital forensics, internet search companies, and language processing software companies.
Team
Group members
Supervisors
Resources
- Standard PC
- MATLAB
- Python
- Reference books
- Takahashi EVT file
- English and foreign texts
<ref> tag;
no text was provided for refs named ARIZ<ref> tag;
no text was provided for refs named Zandbergen1<ref> tag;
no text was provided for refs named Stolfi1<ref> tag;
no text was provided for refs named Zandbergen3<ref> tag;
no text was provided for refs named AUTH