HTERM - Revision history

Jhoger: /* Screen Shots: HTERM to Linux */

2012-04-26T18:45:36Z

Screen Shots: HTERM to Linux

← Older revision		Revision as of 11:45, 26 April 2012
Line 7:		Line 7:
	=== Powerbase Interview via W3M===		=== Powerbase Interview via W3M===
	[[image:W3M_POWERBASE_INTERVIEW.JPG]]		[[image:W3M_POWERBASE_INTERVIEW.JPG]]

			=== HTML Table from Bitchin100 wiki via W3M ===
			[[image:W3M_TABLE.jpg]]

	=== Bitchin100.com Browsed via W3M ===		=== Bitchin100.com Browsed via W3M ===

Jhoger at 18:43, 26 April 2012

2012-04-26T18:43:23Z

← Older revision		Revision as of 11:43, 26 April 2012
Line 2:		Line 2:

	HTERM is a "dumb terminal" program which implements hardware flow control and rudimentary handling of UTF-8 encoding and ANSI escapes.		HTERM is a "dumb terminal" program which implements hardware flow control and rudimentary handling of UTF-8 encoding and ANSI escapes.

			== Screen Shots: HTERM to Linux==

			=== Powerbase Interview via W3M===
			[[image:W3M_POWERBASE_INTERVIEW.JPG]]

			=== Bitchin100.com Browsed via W3M ===
			[[image:W3M_BITCHIN100.JPG]]

			=== Editing a File with Vim ===
			[[image:I_LIKE_VIM_LORES.JPG]]

			=== Reading Mailing List with Mutt ===
			[[image:MUTT_EMAIL.JPG]]

	== Source Code ==		== Source Code ==

Jhoger: /* Unicode Support */

2011-08-09T16:55:41Z

Unicode Support

← Older revision		Revision as of 09:55, 9 August 2011
Line 89:		Line 89:
	So I knew I could do a bucket hash or linear probing on an array with a simple hash function. But I figured it would be fun to do better... could I find a perfect hash function? That is, for every mapped Unicode character, hashing a given 16-bit Unicode value, it would always map to exactly the right entry in my table.		So I knew I could do a bucket hash or linear probing on an array with a simple hash function. But I figured it would be fun to do better... could I find a perfect hash function? That is, for every mapped Unicode character, hashing a given 16-bit Unicode value, it would always map to exactly the right entry in my table.

	I found something called a Pearson's Hash. This relies on a 256-byte lookup table L. In my case, I pick a function, and look up the most signficant and least signicant byte of the Unicode value in the table, and XOR the values I get together. This gives me an index into my Unicode->M100 table. If I get a character u, I calculate L[u/256] XOR L[u AND 255], or L[u>>8] ^ L[u&0xff]. The resulting value is the index, calculated in constant time... no linear search or binary search, just a calculation.		I found something called a Pearson's Hash. This relies on a 256-byte lookup table L. In my case, I pick a function, and look up the most signficant and least signicant byte of the Unicode value in the table, and XOR the values I get together. This gives me an index into my Unicode->M100 table. If I get a character u, I calculate L[u/256] XOR L[u AND 255], or L[u>>8] ^ L[u&0xff]. The resulting value is the array index, calculated in constant time... no linear search or binary search, just a calculation.

	The problem of course is coming up with the look up table L. Nothing I could find said anything about how to make such a table. Left, I guess as "an exercise for the reader." I tried to do it by hand, but ended up with collisions.		The problem of course is coming up with the look up table L. Nothing I could find said anything about how to make such a table. Left, I guess as "an exercise for the reader." I tried to do it by hand, but ended up with collisions (two Unicode values mapping to the same array index).

	So I wrote Perl script to brute force it. The algorithm the Perl script uses is to create an array of the values 0-255. Then it randomly swaps all positions in the table.		So I wrote Perl script to brute force it. The algorithm the Perl script uses is to create an array of the values 0-255. Then it randomly swaps all positions in the table.

Jhoger: /* Unicode Support */

2011-08-09T16:54:15Z

Unicode Support

← Older revision		Revision as of 09:54, 9 August 2011
Line 89:		Line 89:
	So I knew I could do a bucket hash or linear probing on an array with a simple hash function. But I figured it would be fun to do better... could I find a perfect hash function? That is, for every mapped Unicode character, hashing a given 16-bit Unicode value, it would always map to exactly the right entry in my table.		So I knew I could do a bucket hash or linear probing on an array with a simple hash function. But I figured it would be fun to do better... could I find a perfect hash function? That is, for every mapped Unicode character, hashing a given 16-bit Unicode value, it would always map to exactly the right entry in my table.

	I found something called a Pearson's Hash. This relies on a 256-byte lookup table L. In my case, I pick a function, and look up the most signficant and least signicant byte in the table, and XOR the values I get together. This gives me an index into my Unicode->M100 table. If I get a character u, I calculate L(u/256) XOR L(u AND 255), or L(u>>8) ^ L(u&0xff). The resulting value is the index, calculated in constant time... no linear search or binary search, just a calculation.		I found something called a Pearson's Hash. This relies on a 256-byte lookup table L. In my case, I pick a function, and look up the most signficant and least signicant byte of the Unicode value in the table, and XOR the values I get together. This gives me an index into my Unicode->M100 table. If I get a character u, I calculate L[u/256] XOR L[u AND 255], or L[u>>8] ^ L[u&0xff]. The resulting value is the index, calculated in constant time... no linear search or binary search, just a calculation.

	The problem of course is coming up with the look up table L. Nothing I could find said anything about how to make such a table. Left, I guess as "an exercise for the reader." I tried to do it by hand, but ended up with collisions.		The problem of course is coming up with the look up table L. Nothing I could find said anything about how to make such a table. Left, I guess as "an exercise for the reader." I tried to do it by hand, but ended up with collisions.
Line 95:		Line 95:
	So I wrote Perl script to brute force it. The algorithm the Perl script uses is to create an array of the values 0-255. Then it randomly swaps all positions in the table.		So I wrote Perl script to brute force it. The algorithm the Perl script uses is to create an array of the values 0-255. Then it randomly swaps all positions in the table.

	Next, it walks through many further reordered arrangements in an iterative fashion. I made an ad hoc "fitness function" and used something like ~~Lamarkian~~ evolution to evolve a useful arrangement / seek a nearby minimum. Each iteration the script takes the ordering and applies the fitness function to it. If it improves it (decreases its score) then the new ~~order~~ survives into the next ~~generation~~. If it doesn't improve it, then I revert to the previous order for the next iteration.		Next, it walks through many further reordered arrangements in an iterative fashion. I made an ad hoc "fitness function" and used something like Lamarckian evolution to evolve a useful arrangement / seek a nearby minimum. Each iteration the script takes the ordering and applies the fitness function to it. If it improves it (decreases its score) then the new ordering survives into the next iteration. If it doesn't improve it, then I revert to the previous order for the next iteration.

	The fitness function has various weights for different unwanted things. For example, a collision (two Unicode values mapping to the same spot) has a weight.		The fitness function has various weights for different unwanted things. For example, a collision (two Unicode values mapping to the same spot) has a weight.

	Anyway, to my surprise, once I got the Perl script debugged, and fiddled with the weights, it came up with usable look up tables VERY quickly. There are probably some ways of finding look up tables, but so far the Perl script is working. Obviously you can't search through all orderings... thats 256! . So you need to do some kind of search.		Anyway, to my surprise, once I got the Perl script debugged, and fiddled with the weights, it came up with usable look up tables VERY quickly. There are probably some other ways of finding look up tables, but so far the Perl script is working. Obviously you can't search through all orderings... thats 256 factorial (big). So you need to do some kind of search.

Jhoger at 00:15, 8 August 2011

2011-08-08T00:15:34Z

← Older revision		Revision as of 17:15, 7 August 2011
Line 22:		Line 22:

	http://bitchin100.com/files/t200/HTERM.lst		http://bitchin100.com/files/t200/HTERM.lst

			== Loading HTERM ==

			The process for loading HTERM is the same as a typical .CO file.

			[[Loading a typical CO file]]

	== Unicode Support ==		== Unicode Support ==

Jhoger: /* Unicode Support */

2011-08-07T04:42:57Z

Unicode Support

@@ Line 25: / Line 25: @@
 == Unicode Support ==
-HTERM is primarily designed to interface with Linux systems, either directly connected or through a serial to network bridge. Login consoles on Linux generally use ANSI escapes and UTF-8 encoding.
+UTF-8 characters are an encoding of Unicode. Unicode characters are between 8 and 32 bits. It adopts the 7-bit ASCII characters and extends from there.
-UTF-8 is a Unicode "encoding." It is a way of representing Unicode characters in an 8-bit byte format. Unicode itself is an abstraction... characters in Unicode have a range of 32-bits, with no defined wire or in-memory structure. UTF-8 is one such format.
+HTERM supports UTF-8, since these days it is the default character encoding for Linux and the web. Several command line programs of interest, like w3m and mutt generate UTF-8 encoded characters. w3m is a text mode web browser. When it renders HTML tables, it uses the Unicode line drawing glyphs, the character pi, the copyright symbol, etc. Mutt is a email client. When it represents threads, it uses the line drawing characters to represent something like a tree of emails.
-The Model T laptops also have a 8-bit character code. For characters in the range 0-127, the ASCII range, the characters are the same in UTF-8. For characters above 127, there is no defined correlation between Unicode and the Model T.
+In UTF-8, ASCII characters ($00-$7F) are the same as ever. But higher characters are represented as two to six byte strings in the following format:
-HTERM creates such a correlation. HTERM manages converting between UTF-8, an in-memory Unicode representation, and the Model T's displayable character set.
+The first byte in a UTF-8 encoded character has a run of 1's in the highest bits. The length of the run indicates the number of octets comprising the encoded character. This run of 1's is always followed by a zero. The rest of the bits in the first octet are "payload" of the encoding, that is, the most significant bits of the Unicode value being encoded.
-It turns out that most of the Model 100's character set maps to the 16-bit Unicode range. Given that, and the fact that the Model 100 can deal directly with 16-bit values, HTERM only supports mapping of Unicode characters in the 16-bit range. If it receives a character outside of that range, it will treat it as an unmapped character.
+Following octets start with 10, followed by the payload. Octet order of the payload bits, if prefix bits of each octet are discarded, is in MSB->LSB order.

99.6.119.22 at 03:42, 3 August 2011

2011-08-03T03:42:47Z

← Older revision		Revision as of 20:42, 2 August 2011
Line 6:		Line 6:

	Git repo: http://bitchin100.com/pub-git/hterm.git/		Git repo: http://bitchin100.com/pub-git/hterm.git/

			== Latest Build ==

			(No released build with Unicode support yet)

			=== M100/T102 ===

			http://bitchin100.com/files/m10x/HTERM.CO

			http://bitchin100.com/files/m10x/HTERM.lst

			=== T200 ===

			http://bitchin100.com/files/t200/HTERM.CO

			http://bitchin100.com/files/t200/HTERM.lst

	== Unicode Support ==		== Unicode Support ==

99.6.119.22: /* Unicode Support */

2011-07-30T23:31:03Z

Unicode Support

← Older revision		Revision as of 16:31, 30 July 2011
Line 16:		Line 16:

	HTERM creates such a correlation. HTERM manages converting between UTF-8, an in-memory Unicode representation, and the Model T's displayable character set.		HTERM creates such a correlation. HTERM manages converting between UTF-8, an in-memory Unicode representation, and the Model T's displayable character set.

			It turns out that most of the Model 100's character set maps to the 16-bit Unicode range. Given that, and the fact that the Model 100 can deal directly with 16-bit values, HTERM only supports mapping of Unicode characters in the 16-bit range. If it receives a character outside of that range, it will treat it as an unmapped character.

99.6.119.22 at 22:24, 28 July 2011

2011-07-28T22:24:25Z

← Older revision		Revision as of 15:24, 28 July 2011
Line 6:		Line 6:

	Git repo: http://bitchin100.com/pub-git/hterm.git/		Git repo: http://bitchin100.com/pub-git/hterm.git/

			== Unicode Support ==

			HTERM is primarily designed to interface with Linux systems, either directly connected or through a serial to network bridge. Login consoles on Linux generally use ANSI escapes and UTF-8 encoding.

			UTF-8 is a Unicode "encoding." It is a way of representing Unicode characters in an 8-bit byte format. Unicode itself is an abstraction... characters in Unicode have a range of 32-bits, with no defined wire or in-memory structure. UTF-8 is one such format.

			The Model T laptops also have a 8-bit character code. For characters in the range 0-127, the ASCII range, the characters are the same in UTF-8. For characters above 127, there is no defined correlation between Unicode and the Model T.

			HTERM creates such a correlation. HTERM manages converting between UTF-8, an in-memory Unicode representation, and the Model T's displayable character set.

Jhoger: /* Purpose */

2011-07-16T06:38:44Z

Purpose

← Older revision		Revision as of 23:38, 15 July 2011
Line 1:		Line 1:
	== Purpose ==		== Purpose ==

	HTERM is a "dumb terminal" program which implements hardware flow control and ~~reasonable~~ handling of UTF-8 encoding and ANSI escapes.		HTERM is a "dumb terminal" program which implements hardware flow control and rudimentary handling of UTF-8 encoding and ANSI escapes.

	== Source Code ==		== Source Code ==

	Git repo: http://bitchin100.com/pub-git/hterm.git/		Git repo: http://bitchin100.com/pub-git/hterm.git/