• ホーム
• 進行中の研究
  • TEI準拠の中国古代文書簡牘資料庫構築
• 今までの研究
  • 出土簡牘の生態的研究
  • 額済納現地調査2009
  • 額済納現地調査2013
  • 居延漢簡調査
  • 「肩水金関漢簡による漢代西北交通・防衛機構の研究」（青木俊介）
  • 「周縁領域からみた秦漢帝国の総合的研究」（高村武幸）
  • 「新出簡牘資料を用いた戦国秦から統一秦にかけての国制変革に関する研究」（渡邉英幸）
  • 「最新史料に見る秦・漢法制の変革と帝制中国の成立」（陶安あんど）
  • 「中国古代における 家族と「移動」の多角的研究─静態的家族観からの脱却をめざして─」（鈴木直美）
  • 「三国呉・長沙の年齢史─人生の諸段階と同居家族」（鷲尾祐子）
  • 「五一広場東漢簡牘よりみた後漢時代の在地社会」（飯田祥子）
  • 中国古代簡牘の横断領域的研究(1)
  • 中国古代簡牘の横断領域的研究(2)
  • 中国古代簡牘の横断領域的研究(3)
  • 中国古代簡牘の横断領域的研究(4)
  • 中国古代簡牘の横断領域的研究(5)
• 資料庫実験室
  • 趣旨説明
  • 9 TEI試作品――校訂情報02
  • 8 TEI試作品――走馬楼呉簡吏民簿ＴＥＩテキスト化試案
  • 7 TEI試作品――校訂情報02
  • 6 索引データの再分析
  • 5 形態と様式分類のデータ収集
  • 4 DB構想
  • 3 HTML形式の資料庫
  • 2 索引稿の編集
  • 1 ワード形式の訳注稿
• TEI2026
  • Table of Contents
  • 1. The paper’s objective
  • 2. The encoding of traditional markup language
  • 3. Automated text collation
  • 4. The encoding and redefinition of characters
  • 5. References
• 研究仲間
• 史料ノート

TEI2026 Presentation Material

2. The encoding of traditional markup language

2.1. The traditional markup language

In praxis, Chinese paleography is the art of transcribing and collating early Chinese manuscripts. At the outset, paleography was primarily focused on the recognition and comparative analyses in space and time of what were known as pre-Qin characters (Xiānqín wénzì先秦文字), i.e. characters prior to the unification of Chinese characters under the First Emperor of the Qin[7]". However, the abundance of excavated written materials led to an increased interest among paleographers in the development of script during the Western and Eastern Han periods in recent decades. Now, the range of paleographic research reaches down to the late second and early third century CE, when Kǎishū楷書, or "standard script", emerged[8]". This new lower limit of paleographic research approximately overlaps with the end of industrial production and consumption of wooden tablets and bamboo slips as writing materials and their replacement by paper[9]", making Chinese paleography a convenient and indispensable tool for the deciphering of wood and bamboo documents or manuscripts.

To keep track of how modern transcribers interpret the original text, Chinese paleography has developed a distinctive markup language for storing this information. It is without doubt the most essential requirement for any encoding of pertinent texts to retain the information found in paleographic markup. Therefore, its basic content should be briefly introduced here.

Figure 4 shows an excerpt of the transcription of the compilation. The Arabic numbers refer to the serial number of slips in the modern text arrangement. The text is divided into indented paragraphs according to its appearance on the slips, one separate paragraph represents the string of characters found on one slip. The boxes and various brackets dispersed throughout the text are examples of the said paleographic marking language.

Like the brackets and the boxes, punctuation marks also comprise an information layer that reflects the transcribers’ comprehension of the text. In many editions of excavated manuscripts or documents, a highly enriched markup transcription is often accompanied by a more restrained transcription, which is considered closer to the “original” literal appearance of the text. This sort of “literal” or “restrained” transcription of the same text example as in figure 4 looks like figure 5.

In this restrained version, the supplements for omitted or illegible characters remain in place because literal parts of the imagined original text would be absent otherwise. The parenthesized question marks expressing uncertainty are usually maintained too. “●” and “┘” are standardized modern representations of two typical sorts of original punctuation marks[10]". “〼” marks the loss of a slip fragment at the beginning or end of the remaining text fragment. In the enriched version, the lost text was supplemented in black lenticular brackets (“【】”).

2.2. Semantic correlations between traditional markup and TEI-elements

As the markup language reflects the transcribers’ comprehension of a particular text, the usage can differ slightly across different transcriptions. For a responsible transcription, it is essential to give an overview of the marking elements. Figure 6 shows the legend used in my edition of the compilation.

It is astonishing how neatly the semantic meaning of these elements of the traditional markup language overlaps with well-known elements and attributes in the TEI-guidelines. Therefore, a comparison like the table 1 may be more eloquent and helpful for the non-Sinophile audience than individual explanations by the author.

In recent transcriptions, a few of these markup symbols tend to be confused. A common source of confusion involves the information linked to the tortoise shell brackets ("〔〕")[12]". They should be strictly distinguished from square brackets (“［］”), but they get intermingled often so that tortoise shell brackets or angle brackets are used to express corrections to alleged misspellings and to point to redundant characters. Another common conflation is the application of black lenticular brackets ( “【】”) to different sorts of uncertainty, ranging from doubts in the transcription to context-based supplementary transcription of characters lost because of surface damage, etc. Of course, these confusions lead to huge losses of transcription information and should, for the purpose of best practice, be avoided.

2.3. Basic Thoughts on automated recognition and encoding

As the discovery of Early Chinese written materials started right from the beginning of the 20th century, an abundance of transcriptions of wooden tablets and bamboo slips has been accumulated since then and is waiting to be properly encoded. Manual encoding would indeed prove to be a grueling task that is likely to be marred by errors. An automated way of recognizing and encoding the traditional markup language is therefore strongly desirable.

In my view, old-fashioned regular expressions are the most reliable tool for the recognition part of the task, and almost any programming language would easily do the job of rendering the recognized markups into TEI-elements. In theory, a reliable text file version of the original text[13], a list of regular expressions capable of matching the various markup symbols, and some basic programming skills are all that we need to accomplish this task.

The basic flow of a script for automated recognition and encoding could be imagined as follows: The script processes each line of the original transcription sequentially, with each line correlating to the sequence of characters carried by a single slip in the original text. For each line, the script hovers over the text character by character and checks for markups connected with the specific character by applying a list of regular expressions. If markups are detected, the script chops off both the character and its markups, then assigns encoding to a corresponding subroutine which outputs the encoded form of the character. Otherwise, only the first character is chopped off without any further modification. After each chopping process, the pertinent character is added to a <p> element that holds the serial number of the slip carrying the character as an @xml:id attribute[14]. The flow chart appears as figure 7.

Table 2 shows the regular expressions that could match the markups listed in table 1. The only markup that cannot be accurately represented in text format is the “字”, an illegible character supplied by the transcriber based on context information, indicated by a box around the supplied character. In the experiments with automated recognition, this markup was substituted with a postposed two-byte ampersand (“＆”)[15].

2.4. Practical obstacles to automatization

In practice, some peculiarities made finding the proper regular expressions and their subsequent processing more complicated than anticipated initially. First, the abbreviation markers found in the original texts (“﹦”) are used as a shorthand for the replication or duplication of both characters and character components. The former is named chóngwén重文(character duplication), and the latter héwén合文( =character composition, compound character) in modern Chinese terminology.

Table 1 in the previous section listed the compound character “粟﹦”, which is a composition of the abbreviation marker “﹦” with the character “sù粟(millet/ grain)”. The character “sù粟” entails the character “mǐ米(rice)” as a building component, the abbreviation marker indicating the duplication of this character component. Since the component is replicated as an independent character, the expansion stands for the compound word “sùmǐ粟米(threshed grain or rice).

The challenging aspect of compound characters lies in the random positioning of duplicated character components. While the duplicated component “mǐ米” in “sùmǐ粟﹦” is postposed, the component “dà大(large)” in the compound character “夫﹦” is supposed to be preposed, resulting in the compound word “dàfū大夫(denomination of an official or social rank)”. The expansion of the abbreviation “sùmǐ粟﹦” leads to an “AB” structure, whereas the abbreviation “dàfū夫﹦” turns into “BA”, with “A” representing the original character originally and “B” for the editorial expansion. Table 3 illustrates the same difference by translating the abbreviation and its expansion into TEI-elements.

The character duplication (chóngwén重文) listed in Table 1 is “guǐ癸﹦děng等﹦”. “Guǐ” is the personal name of a culprit, “děng” meaning “others/ et al.”. The length of the duplication expression can range from duplication of a single character (“A﹦”) to the duplication of three to four characters (“A﹦B﹦C﹦D﹦”), but the expansion of such duplication expressions is quite regular in general, “A﹦”, “A﹦B﹦”, “A﹦B﹦C﹦”, “A﹦B﹦C﹦D﹦” resulting in “AA”, “ABAB”, “ABCABC”, “ABCDABCD” respectively. However, the expansion text is prone to entail multiple modern punctuation marks, whose distribution appears to be more or less arbitrary.

The abbreviation example “guǐ癸﹦děng等﹦” in Table 1 has been taken from slip 6 of the compilation. The expansion reads “guǐděng癸等。guǐděng癸等”. The first “guǐděng” was understood as the object to the predicate of the preceding sentence and the second “guǐděng” as signaling the subject of the following sentence. Thus, both were separated by a period (“。”). Not only the sort of punctuation mark separating the original phrase from the editorial expansion can vary depending on the context, or the transcriber’s understanding of the context, there can also appear punctuation marks within both the original string and the editorial expansion. For example, slip 134 of the compilation holds the expression “sì肆﹦shì室﹦”, “sì” and “shì” meaning “store, shop” and “house” respectively. As “sìshì” itself is a list of two nouns that need to be separated by a comma, the entire expression turns into something like “sì肆﹦shì室﹦（sì肆、shì室。sì肆、shì室）”, entailing three modern punctuation marks in total.

Another aspect adding complexity to automated traditional markup processing involves the accumulation of various markups associated with a single character in the original text. The most common combination of marks is the inclusion of the uncertainty marker “（？）” in other markups. In such cases, the parentheses around the two-byte question mark (“？”) are omitted. For instance, the character “yǐ已(=already)” is written in the shape of the modern character “sì巳” on slip 121 of the said compilation. Without any uncertainty, this would result in the transcription “巳（已）”, an equivalent to the TEI element “<choice><orig>巳</orig><reg>已</reg></choice>”. Since there remain some doubts about the deciphering of the original shape “巳”, a question mark was added in front of the regularized writing form “已”, which turned the entire transcription of this single character into “巳（？已）”. Likewise, the transcription “詣﹦（？詣，詣）” in slip 158 expresses uncertainty about the transcription of the abbreviation marker “﹦”.

The position of the uncertainty marker can also change to a postposition to the regularized writing form, as in “臞（虞？）” on slip 166. Before this appearance, it has been unknown that the character “yú虞(= to scheme, transferred meaning deriving from to plan)” could be written in the shape of “臞”, which ordinarily is read “qú (=emaciated)”. In the first edition of the compilation from 2013, the author did not dare to regularize “qú臞” into “yú虞” and rendered the entire character as a gap (“□”), based on the fact that a small part of the shape “臞” was incomplete. After extensive deliberation, the author realized that he could not continue dodging the recognizable shape “臞” of this character and, based on the context, rendered it tentatively as the character “yú虞”[18].

Within the transcription of the compilation, instances of inclusion of the uncertainty marker are limited to regularizations and renderings of abbreviation marks. However, uncertainty markers can appear in almost all other markup types, and pertinent instances are found in other excavated literature and in excavated documents. An example from the administrative documents discovered in an abundant well at the Liye-site, Hunan province, might suffice. On tablet 5-31, the character “suǒ所 (=place)” appears to have been mistakenly written as a “jīn斤 (= a unit of weight)”. Because the fragmentary context doesn’t provide sufficient proof, uncertainty remains, which could be rendered as “斤〔所？〕”, equivalent to “<choice><sic>斤</sic><corr>所<certainty locus="value" degree="0.6"/></corr></choice>”.

Moreover, the aggregate usage of various markups extends beyond just incorporating uncertainty indicators. The regularization marker is often joined with other markers through hyphenation. For instance, the shape “𣄵” on slip 143, which usually is read “liàng (=expression of despise against negative events), is the mistaken form of the character “jiù就”, which in this case should be rendered into the regularized form “jiù僦”, as it expresses the meaning of “to hire”. The corresponding transcription is “𣄵〔就-僦〕”, which is equivalent to the TEI-element “<choice><orig><choice><sic>𣄵</sic><corr>就</corr></choice></orig><reg>僦</reg></choice>”.

Another example is the transcription “言（音-意）” on slip 145. In this instance, “yì意” functions as a verb carrying the meaning “to intend”. In paleographic sources, this verb is often written in the shape of “音”, which is ordinarily read “yīn (=sound)”. To render the shape “音” into the character “yì意”, the typical transcription would be “音（意）”. In this special case, however, we witness the rare phenomenon that the shape “音” is replaced by the etymologically akin shape “言”, which as a modern character would be read “yán (=to speak)”. Given the close association between “音” and “言” in early Chinese writtings, the usage of the shape “言” cannot be regarded as an error. Rendering of “言” into “音” ought to be viewed instead as a kind of regularization within the framework of early character usage. Put differently, the stated transcription constitutes a dual regularization. The first regularization renders the rare ancient form of “言” into the commonly seen ancient form of “音”; the second regularization turns the ancient form of “音” into the modern character “yì意”. The corresponding TEI-element would be “<choice><orig><choice><sic>言</sic><reg>音</reg></choice></orig><reg>意</reg></choice>”.

Regularization is also coupled with the expansion of abbreviations. For example, slip 185 carries the string “气﹦鞫﹦”. Merely expending the abbreviation would result in the transcription “气﹦鞫﹦（气鞫，气鞫）”. However, “气”, usually read “qì (=gas, air) is the early shape for the modern character “qǐ乞 (to beg)”, regularization of which should be transcribed as “气（乞）”. The combination of the two markups results in “气﹦鞫﹦（气-乞鞫，气-乞鞫）[19]”, an equivalent to the TEI-element “<choice><abbr>气<am>﹦</am>鞫<am>﹦</am></abbr><expan><choice><orig>气</orig><reg>乞</reg></choice>鞫， <choice><ex>气</ex><reg>乞</reg></choice><ex>鞫</ex></expan></choice>”.

Generally speaking, the aggregate usage of different markups merely complicate the regular expressions required to accurately identify the traditional markup language used in transcriptions. By contrast, the inclusion of uncertainty markers within expansions of abbreviations can result in ambiguities that necessitate manual intervention. Given that the parentheses are omitted in the such cases of inclusion, the shape of the uncertainty marker blends indistinguishable with the modern question mark, both being written in the two-byte form “？”. Thus, the question mark in the expression “別﹦直﹦（別直？別直）” could easily be misinterpreted as an uncertainty marker.

Similar ambiguities arise in cases where original punctuation marks are standardized into modern question marks. On slip 82 a question is divided from the following sentence by an original punctuation mark in the shape of “┘”. Separation marks of this kind were employed to separate single words, clauses, or even entire sentences. Accordingly, the rendering into modern Chinese punctuation marks as part of a regularization can vary from two different forms of commas, “，(dòu hào逗號)” and “、(dùn hào頓號)”, to periods (“。jù hào句號”)，and to question marks or exclamation marks (“？wèn hào文號”, “！gǎntàn hào感嘆號”), etc[20] . Sometimes, the said separation mark is used in situations where modern Chinese punctuation is not necessary. In such cases, it would be merely transcribed as “┘”. Hence, the transcription “┘（？）” on slip 82, which actually is the result of a regularization, cannot be discerned from an uncertainty marker “（？）” put behind the original punctuation mark “┘”.

Furthermore, the cumulative use of markups within the supplement marker “【】” requires some adjustment within the flow chart. The slips 17, 39, and 223 contain the following supplements for text fragments lost by damage to the slips:

【萬六百卌（四十）錢。……】（17）

【盜﹦（盜盜）殺傷好】（39）

【癸﹦（癸，癸）攺（改）】（223）

In all the three cases, the supplement is indicated by the supplement marker “【】”. Other markups appear in a seemingly arbitrary distribution among the supplied characters. Regularizations ( “卌（四十）” and “攺（改）”) are found for the fourth and the third characters of slip 17 and 223 respectively; abbreviations (“盜﹦（盜盜）” and “癸﹦（癸，癸）”) at the beginning of the supplement to slip 39 and 223, and a gap of unknown quantity (“……”) at the end of the supplement to slip 17.

At least two approaches exist for dealing with these aggregated markups. One method is to recursively input the supplied text fragment as an independent text line. After catching the supplements “萬六百卌（四十）錢。……”, “盜﹦（盜盜）殺傷好”, and “癸﹦（癸，癸）攺（改）” with the help of the regular expression r'^【([^】]+)】', the script hovers over the text fragment character by character, checks for markups, and chops off characters and markup expressions, just as shown in figure 7 for ordinary text lines. The sole distinction is that the encoded output is not stored directly in a final <p> element but is instead returned to the sub-routine that handles the encoding of supplements. Only after the encoding of the entire output as a supplement is it finally added to the <p> element.

Depending on the script’s design, recursive input and the handling of its output might entail certain challenges. If so, an alternative way to handle it exists. The regular expression r'^【([^】]+)】' is divided into the two separate expressions r'^【' and r'^】'. Every time the first expression matches, a flag is activated, instructing all following sub-routines to direct their output into a temporary variable. Each time the second expression matches, the flag is reset, and the temporary variable’s contents are passed to a subroutine responsible for the concluding encoding and storage of the supplement.

2.5. Preliminary experiments with automat

Following several attempts to surmount the practical challenges noted previously, an initial script was crafted to convert traditional transcriptions of Chinese excavated literature into TEI-conform XML files. The entire script and related files can be freely downloaded from a Japanese website maintained by the author and collaborating Japanese colleagues[21].

The regular expressions that were finally used to match and recognize traditional markups are listed in table 4.

The script was applied to the text of a revised edition from 2021 of the compilation. The text was stored in a text file named “Igokutojo(Zenjoho)02.txt” , whose appearance basically looks like figure 8[23].

The script's output is an XML file in TEI-conformant format, with TEI-elements embedded directly within the main text body[24]. Figure 9 displays the initial portion of the main text.

To ensure the accuracy of the encoding, XSL-Transformations were employed to revert the output file into two distinct text documents. One contains the full transcription including all modern regularizations, corrections, and supplements; the other contains a simplified transcription without regularizations etc. that comes closer to the original appearance of the text. Figure 10 displays the original "enriched" transcription[25] that aligns with the version I incorporated into the infrared slip images in the first edition of the compilation.

Figure 11 illustrates the latter "restrained" transcription. It aligns with the version that was incorporated into the color images of the slips from the initial edition published in 2013[26].

As the output of text information can be easily controlled by XSL transformation, a wide range of applications open up. For example, limiting the output to regularizations results in a list of characters that conventionally have been categorized as phonetic loan characters (tōngjiǎ zì通假字)[27], like figure 12[28].

The automated encoding was programmed using Python. Python is a highly object-oriented programming language, while XSLT is strongly influenced by functional oriented languages, which makes it work completely differently. The author must confess that he had a much harder time getting accustomed to using templates matching xml-elements than when starting to learn Python a few years earlier. The more the outcome of the XSL-transformation confesses. The consistent results across different programming languages prove the machine's readability and the reliability of the recognition systems.