CS520
Spring 2013
Program 1
Due Sunday February 10

Write a C program, decodeUTF8, that will read a file containing UTF-8 encodings of a sequence of Unicode characters, decode the UTF-8 to determine the Unicode characters, and write those decoded characters in UTF-32 format to an output file. The UTF-32 characters should be written to the file in Little Endian format.

The program will take two arguments: the name of the input file as the first argument and the name of the output file as the second argument. The program will return a status of -1 if the file is not a valid UTF-8 file; othewise it will return a status of 0. Put all your source code in the file decodeUTF8.c.

An empty input file is okay. You should simply produce an empty output file.

A file is a valid UTF-8 file if it does not contain any of these errors:

• An overlong encoding.

• An unexpected continuation byte.

• A start byte not followed by enough continuation bytes.

• A 4-byte sequence starting with 0xF4 that decodes to a value greater than 0x10FFFF.

• A sequence that decodes to a noncharacter.

• A sequence that decodes to a value in the range 0xD800-0xDFFF.

fprintf is a generalization of printf that prints to a particular stream (rather than always to stdout). For example:

fprintf(stderr, "unexpected continuation byte at offset %d\n", offset);

You may exit the program after reporting the first error.

You should write other programs to create interesting test cases.

To complete this program, you need to understand the following C features:

• hexadecimal constants, such as 0x7F

• left shift, such as x << 6, which will produce a value by left shifting the value of x six bit positions, filling with zeros on the right and discarding the bits that go off the end on the left. (Note that x is not modified, rather an intermediate value is produced, similar to what happens when computing x + 1.) Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = x << 3;
  

  y will be assigned 0x18.

• right shift, such as x >> 6, which will produce a value by right shifting the value of x six bit positions. If the value being shifted is of unsigned type, then the result is filled with zeros on the left. If the value being shifted is of signed type, then the C compiler decides whether to fill with zeros, or fill by replicating the sign bit. (In either case, the bits that go off the end on the right are discarded.) In order to write portable code you should avoid right shifting a signed type.

  Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = x >> 3;
  

  y will be assigned 0x14.

• bitwise OR, which produces a 1 at a bit position if at least one of the input operands has a 1 at that position; otherwise 0 is produced at that bit position. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = 0x32;
  unsigned char z = x | y;
  

  z will be assigned 0xB3.

• bitwise AND, which produces a 1 at a bit position if both input operands have a 1 at that position; otherwise 0 is produced at that bit position. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = 0x32;
  unsigned char z = x & y;
  

  z will be assigned 0x22.

• bitwise XOR (exclusive OR), which produces a 1 at a bit position if one of the input operands has a 1 at that position and the other input operand has a 0 at that position; otherwise 0 is produced at that bit position. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = 0x32;
  unsigned char z = x ^ y;
  

  z will be assigned 0x91.

• bitwise complement, which produces 1 at a bit position if the input operand has a 0 at that position and produces 0 at a bit position if the input operand has a 1 at that position. x. Consider this example:

  unsigned char x = 0xA3;
  unsigned char y = ~x;
  

  y will be assigned 0x5C.

unsigned int c = getchar();
if (c == EOF)
{
  fprintf(stderr, "unexpected EOF!\n");
}
else if ((c >> 4) == 0x0E)
{
  printf("c contains a three-byte UTF-8 sequence with");
  printf(" %02x for the data bits\n", c & 0x0F);
}

Or, to take the two bytes from a two-byte UTF-8 sequence, extract the data bits, and then combine the data bits to form the UTF-16 character:

unsigned int convertTwoByteUtf8ToUtf16(unsigned int byte1,
                                       unsigned int byte2)
{
  if ((byte1 >> 5) != 0x06)
  {
    fprintf(stderr, "byte1 does not contain UTF-8 two-byte start byte!\n");
    return 0;
  }

  if ((byte2 >> 6) != 0x02)
  {
    fprintf(stderr, "byte2 does not contain UTF-8 continuation byte!\n");
    return 0;
  }

  return (((byte1 & 0x1F) << 6) | (byte2 & 0x3F));
}

1. 60 points will be awarded for properly handling one-byte UTF-8 encodings.
2. 10 additional points will be awarded for also handling two-byte encodings.
3. 10 additional points will be awarded for also handling three-byte encodings.
4. 10 additional points will be awarded for also handling four-byte encodings.
5. 10 additional points will be awarded for properly detecting errors in the input file.

In addition, remember, you may lose points if your program is not properly structured or adequately documented. Coding guidelines are given on the course overview webpage.

Your programs will be graded using agate.cs.unh.edu so be sure to test in that environment.

Your programs should be submitted for grading from agate.cs.unh.edu. To turn in this program, type:

% ~cs520/bin/submit prog1 decodeUTF8.c

Submissions can be checked by typing:

% ~cs520/bin/scheck prog1

This assignment is due Sunday February 10. The standard late policy concerning late submissions will be in effect. See the course overview webpage.

Remember: as always you are expected to do your own work on this assignment. Copying code from another student or from sites on the internet is explicitly forbidden!

Comments and questions should be directed to hatcher@unh.edu