CS520
Spring 2015
Program 1
Due Sunday, February 8

Write two C programs:

• utf8to16.

  The program reads a file containing UTF-8 encodings of a sequence of Unicode characters, decodes the UTF-8 to determine the Unicode characters, and writes those decoded characters in UTF-16 format to an output file. The UTF-16 characters should be written to the file in Little Endian format.

  The program will take two arguments: the name of the input file as the first argument and the name of the output file as the second argument. Print an appropriate error message if the input file cannot be opened for reading or the output file cannot be opened for writing. (If the user specifies the same file for both input and output, then the behaviour is undefined, meaning that there is no particular requirement for how you handle this case.)

  An empty input file is okay. You should simply produce an empty output file.

  A file is a valid UTF-8 file if it does not contain any of these errors:

  • An overlong encoding.

  • An unexpected continuation byte.

  • A start byte not followed by enough continuation bytes.

  • A 4-byte sequence starting with 0xF4 that decodes to a value greater than 0x10FFFF.

  • A sequence that decodes to a noncharacter.

  • A sequence that decodes to a value in the range 0xD800-0xDFFF (i.e. a leading or trailing surrogate).

  If an error is detected, print an appropriate error message to stderr that includes the offset in the file for the start byte for the sequence that is in error. In the case of an unexpected continuation byte, print the offset of that byte.

  You may exit the program after reporting the first error.

  The program will return a status of -1 if an error is encountered; otherwise it will return a status of 0.

  Put all your source code in the file utf8to16.c.

• utf16to8.

  The program reads a file containing UTF-16 encodings of a sequence of Unicode characters, decodes the UTF-16 to determine the Unicode characters, and writes those decoded characters in UTF-8 format to an output file. The UTF-16 characters will be stored in the input file in Little Endian format.

  The program will take two arguments: the name of the input file as the first argument and the name of the output file as the second argument.

  An empty input file is okay. You should simply produce an empty output file.

  A file is a valid UTF-16 file if it does not contain:

  • A value that denotes a noncharacter.

  • Unpaired surrogates: A value in the range 0xD800 to 0xDBFF not followed by a value in the range 0xDC00 to 0xDFFF, or any value in the range 0xDC00 to 0xDFFF not preceded by a value in the range 0xD800 to 0xDBFF.

  If an error is detected, print an appropriate error message to stderr that includes the offset in the file for the start byte for the UTF-16 value that is in error.

  You may exit the program after reporting the first error.

  The program will return a status of -1 if an error is encountered; otherwise it will return a status of 0.

  Put all your source code in the file utf16to8c.

fprintf(stderr, "unexpected continuation byte at offset %d\n", offset);

You should write other programs to create interesting test cases.

To complete this assignment, you need to understand the following C features:

• hexadecimal constants, such as 0x7F

• left shift, such as x << 6, which will produce a value by left shifting the value of x six bit positions, filling with zeros on the right and discarding the bits that go off the end on the left. (Note that x is not modified, rather an intermediate value is produced, similar to what happens when computing x + 1.) Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = x << 3;
  

  y will be assigned 0x18.

• right shift, such as x >> 6, which will produce a value by right shifting the value of x six bit positions. If the value being shifted is of unsigned type, then the result is filled with zeros on the left. If the value being shifted is of signed type, then the C compiler decides whether to fill with zeros, or fill by replicating the sign bit. (In either case, the bits that go off the end on the right are discarded.) In order to write portable code you should avoid right shifting a signed type.

  Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = x >> 3;
  

  y will be assigned 0x14.

• bitwise OR, which produces a 1 at a bit position if at least one of the input operands has a 1 at that position; otherwise 0 is produced at that bit position. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = 0x32;
  unsigned char z = x | y;
  

  z will be assigned 0xB3.

• bitwise AND, which produces a 1 at a bit position if both input operands have a 1 at that position; otherwise 0 is produced at that bit position. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = 0x32;
  unsigned char z = x & y;
  

  z will be assigned 0x22.

• bitwise XOR (exclusive OR), which produces a 1 at a bit position if one of the input operands has a 1 at that position and the other input operand has a 0 at that position; otherwise 0 is produced at that bit position. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = 0x32;
  unsigned char z = x ^ y;
  

  z will be assigned 0x91.

• bitwise complement, which produces 1 at a bit position if the input operand has a 0 at that position and produces 0 at a bit position if the input operand has a 1 at that position. x. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = ~x;
  

  y will be assigned 0x5C.

unsigned int c = getchar();
if (c == EOF)
{
  fprintf(stderr, "unexpected EOF!\n");
}
else if ((c >> 4) == 0x0E)
{
  printf("c contains a three-byte UTF-8 sequence with");
  printf(" %02x for the data bits\n", c & 0x0F);
}

Or, to take the two bytes from a two-byte UTF-8 sequence, extract the data bits, and then combine the data bits to form the UTF-16 character:

unsigned int convertTwoByteUtf8ToUtf16(unsigned int byte1,
                                       unsigned int byte2)
{
  if ((byte1 >> 5) != 0x06)
  {
    fprintf(stderr, "byte1 does not contain UTF-8 two-byte start byte!\n");
    return 0;
  }

  if ((byte2 >> 6) != 0x02)
  {
    fprintf(stderr, "byte2 does not contain UTF-8 continuation byte!\n");
    return 0;
  }

  return (((byte1 & 0x1F) << 6) | (byte2 & 0x3F));
}

1. 60 points will be awarded for properly converting one-byte UTF-8 encodings to and from UTF-16 encodings.
2. 10 additional points will be awarded for also converting two-byte UTF-8 encodings to and from UTF-16 encodings.
3. 10 additional points will be awarded for also converting three-byte UTF-8 encodings to and from UTF-16 encodings.
4. 10 additional points will be awarded for also converting four-byte UTF-8 encodings to and from UTF-16 encodings.
5. 10 additional points will be awarded for properly detecting errors in the input files.

In addition, remember, you may lose points if your program is not properly structured or adequately documented. Coding guidelines are given on the course overview webpage.

Your programs will be graded using agate.cs.unh.edu so be sure to test in that environment. Your programs will be compiled using these gcc flags: -g -Wall -std=c99.

Your programs should be submitted for grading from agate.cs.unh.edu. To turn in this program, type:

% ~cs520/bin/submit prog1 utf8to16.c utf16to8.c

Submissions can be checked by typing:

% ~cs520/bin/scheck prog1

This assignment is due Sunday February 8. The standard late policy concerning late submissions will be in effect. See the course overview webpage.

Remember: as always you are expected to do your own work on this assignment. Copying code from another student or from sites on the internet is explicitly forbidden!

Comments and questions should be directed to hatcher@unh.edu