Projects:2014S1-44 Cracking the Voynich Manuscript Code
The Voynich Manscript is a mysterious 15th century book that no one today know what it says or who wrote it. The book is in a strange alphabet. See details here.
Fortunately the whole book has been converted into an electronic format with each character changed to a convenient ascii character. We want you to write software that will search the text and perform statistical tests to get clues as to the nature of the writing. Does the document bear the statistics of a natural language or is it a fake?
We already have Support Vector Machine (SVM) and Multiple Discriminant Analysis (MDA) software that you can adapt for your purposes. This software is set up to test if two texts are written by the same author or not. The great thing about our software is that it is independent of language. So you could compare it against the existing writings of Roger Bacon, who is a suspected author
Contents
Project information
Specific tasks
- Phase 1: Characterize the text. Write scripts that count its features. How many words? How long is the alphabet? Word frequencies? Probability of one letter following another. Probability of two letter pairs (2-grams) and n-letter group (n-grams). Compare these in a table with known languages obtained by running your same code on the Declaration of Human Rights. Don't forget to get a short paragraph of English and manually count everything and then run it on your code to cross check it is counting correctly. You must always validate your code or you will lose marks.
- Phase 2: Write a general descriptor for each picture in the book, eg. water, woman, tree, flower, vegetable, leaf, dancing etc. Associate each descriptor with the appropriate paper. Write some code to find which words on a page are unique to those pages with those descriptors. Which words also suddenly increase in frequency on those pages with shared descriptors? Tabulate the results.
- Phase 3: Investigate the use of Word Recurrence Interval (WRI) versus rank plots. Plot WRI curves of the Voynich versus other languages from the Declaration of Human Rights.
- Phase 4: Think up some other ideas to try out.
- Phase 5: As WRI is a language-independent metric, you can select classification features based on WRI. Then you can run an SVM and an MDA classifier to compare the Voynich against other languages in the Declaration of Human Rights. Then you can run it against the works of specific authors of interest such as Roger Bacon, John Dee, and Edward Kelley.
Deliverables and Progress
- Proposal seminar
- Progress report
- Final seminar
- Final report
- Poster
- Project exhibition 'expo'
- YouTube video
- Weekly progress
Follow this link for current progress
End results
It will familiarize you with techniques in information theory, probability, statistics, encryption, decryption, signal classification, and datamining. It will also improve your software skills. The new software tools you develop may lead to new IP in the areas of datamining, automatic text language identification, and also make you rich/famous. The types of jobs out there where these skills are useful are in computer security, comms, digital forensics, internet search companies, and language processing software companies.
Team
Group members
Supervisors
Resources
- Standard PC
- MATLAB
- Python
- Reference books
- Takahashi EVT file
- English and foreign texts
