Learning and working with ECMAScript Standardization Tests

Recently, I’ve learned about browser wars and how standardization for the web came to be. The primary focus is on ECMAScript, another name for JavaScript. If you’re interested in the history of JavaScript and how it got re-branded as ECMAScript, find out more about it here. The summary is that while Brendan Eich had created a compelling language for the web, his implementation rests with the Netscape browser, which at the time was closed source. Other browsers such as Microsoft’s Internet Explorer, desiring the same capabilities reverse engineered Brendan’s work and created their own implementation, calling it JScript to avoid trademark disputes. This means the behavior of JavaScript among browsers was not consistent. Web designers often resorted to multiple hacks to ensure cross browser compatibility. ECMA is an European organization that standardizes technology to which Brendan approached to standardize JavaScript to avoid further fragmentation and to prevent Microsoft from setting their own standards. Since the standardization of the language, software vendors and internet citizens like you and I are now able to contribute to the discussion regarding standards that should be enforced in all browsers. In order for browser vendors to ensure their browser conforms to the standard’s specifications, automated testing was created. It is open to the public such that they can also contribute by providing suggestions and feedback regarding the standard, or help write tests which browser vendors will check their implementations against. I choose to practice writing some tests for the browser vendors.

Obtaining the tests

ECMAScript has a GitHub repository, which keeps track of all tests written for ECMAScript and manages collaboration between all people working standardization. I began there.

Step 1: Read ReadMe.md

Often, I get very excited about working on an open source project that I completely skip README.md hoping to jump right into the heart of the project. The README.md on the tc39/test262 repository is very informative and is a great place to begin reading.

In particular, the runners & harnesses section is the most important. A harness in their terms means a command line software that will execute the tests written for the standard on the vendor’s browsers.

The version of harness I choose was Node.JS since I have the most experience working with it as of late.

Step 2: Install the harness

Following the commands they’ve suggested, I’ve executed the following on the terminal:

git clone https://github.com/tc39/test262.git --depth 1
cd test262
npm install -g test262-harness

Take note, the --depth 1 part of the git command is an option that specifies obtaining the latest commit only. This is useful if our purpose is to obtain the latest version of the test for conformance testing purposes.

Step 3: Execute tests

It is always a good idea to run the unit tests of a project before beginning to work on it. This way, we have an idea if there are any existing errors before introducing code of our own.

test262-harness test/**/*.js

Test results

Using Node.JS version 9.6.0, the 205 tests completed with 199 passed and 6 failed.

The problems for which Google’s V8 engine failed at seem to be related to localization and the handling of the property descriptor in its tests of the harness itself.

Background information: Node.JS is a JavaScript run-time environment built based on Google’s V8 JavaScript engine built for back end processes such as hosting a website or web service.

Step 4: Read CONTRIBUTE.md

Now that the harness is setup and tested to work, we begin by reading CONTRIBUTE.md on the repository.

After reading this document, I feel I’ve understood more about
• Tests naming conventions
• Tester file format and commenting
• How to write tests that runs on the harness

Step 5: Read the Specification

Before writing any tests, we must understand the specifications of our point of interest. The goal is to understand the implementation of Array.prototype.reverse().

This process took me a while as I had to dig deep to understand the intent of the pseudo code as well as the behavior of the abstract functions it referenced.

Step 6: Write some tests for Array.prototype.reverse()

Now that we’ve read CONTRIBUTING.md, it’s time to practice writing some tests. I’ve created array-with-primitives.js as the name of my test file. My goal was to test the following cases:

• The reversal of [] is []
• A collection of numbers is reversed correctly: [1,2] > [2,1]
• A collection of characters is reversed correctly: [‘a’,’b’] > [‘b’, ‘a’]
• A collection with mixed primitive data types should be reversed correctly: [‘a’, 1, undefined, null] > [null, undefined, 1, ‘a’]

As I experimented with writing these tests, I ran into a few problems

1) Testing the case of reversing []

The test I wrote initially was

let a = [];
let reverse = a.reverse();
a,reverse, `#1: a = []; a.reverse(); reverse.length === 0. Actual: ${reverse}`

The specification noted that an object is returned and therefore I assumed a and reverse were two seperate objects to which I could use assert.sameValue() to compare.

What I’ve found over time was that this is not the case. In fact, a also gets modified even as the object is returned and referenced by reverse. After reading the specification in more detail, I realized this behavior is intended. I was able to test this by using the Node environment, then performing

let a = [];
reverse = a.reverse();

The result of this was that both a and reverse had [1] as the result. This is when I realized they are references and cannot be used to check and see if the array is truly empty. I then thought of comparing it by using

reverse,[], `#1: a = []; a.reverse(); reverse.length === 0. Actual: ${reverse}`

This did not work because assert.sameValue() is checking the memory address for which the two variables were pointing to. The solution I came up with to guarantee that it is indeed what I expected was to use the length property of reverse to ensure the length is 0.

2) Checking values inside an array with length > 0

This problem is related to the previous one. My assumption was that I could check the contents of an array and make comparisons by passing in a separate array with the expected result.

a = [1, 2];
reverse = a.reverse();
a, reverse, `#2: a = [1, 2]; a.reverse() !== a. Actual: ${reverse}`

The result was the following

FAIL array-with-primitives.js (default)
#2: a = [‘a’, ‘b’]; a.reverse() !== a. Actual: 2,1 Expected SameValue(«2,1», «2,1») to be false

FAIL array-with-primitives.js (strict-mode)
#2: a = [‘a’, ‘b’]; a.reverse() !== a. Actual: 2,1 Expected SameValue(«2,1», «2,1») to be false

This failure indicated to me at the same time that when the specification mentioned Set() that the current values of the array were being modified. Meaning a's contents were being directly changed when a swap occurs. This makes sense since the specification takes this as an argument to be made into an object. I’ve also realized after that when the object is return, this is similar to C++’s return this; statement, in which the address to the pointer that points to the underlying array in memory is being returned.

This has been confirmed when I investigated into SameValue(). SameValue(x,y) in this scenario should return SameValueNonNumber(x,y). SameValueNonNumber(x,y) states that if x and y are the same Object value return true. Otherwise return false. This further confirms my theory to be true.

However, there’s a second question that came to me. At a quick glance, I thought to myself, “Why did I fail 2 tests when clearly only one has failed?” The reason is because there are two tests that are being executed by the harness. The first test is executed in default mode. the second test is executed in strict-mode. Strict mode is a subset of JavaScript that removes all the features that are the typical leading cause to bugs.

The solution I have to this memory problem was to test each individual element separately.

a = [‘a’, 1, undefined, null];
reverse = a. reverse();
reverse[0], null, `#4: a = [‘a’, 1, undefined, null]; reverse = a.reverse(); reverse[0] !== null. Actual: ${reverse[0]}.`
reverse[1], undefined, `#4: a = [‘a’, 1, undefined, null]; reverse = a.reverse(); reverse[1] !== undefined. Actual: ${reverse[1]}.`
reverse[2], 1, `#4: a = [‘a’, 1, undefined, null]; reverse = a.reverse(); reverse[2] !== 1. Actual: ${reverse[2]}.`
reverse[3], ‘a’, `#4: a = [‘a’, 1, undefined, null]; reverse = a.reverse(); reverse[3] !== ‘a’. Actual: ${reverse[3]}.`

This worked, but it was tedious and clumsy.

3) When a test fails, all tests thereafter do not get executed.

This was a behavior that I did not expect. Considering the harness is a tool to check for the browser’s level of compliance, failing one tests shows “2 tests failed”. However, what confused me was that even after inducing 4 failed tests, the result was 2 failed tests. This is how I realized testing stops the moment a test fails. I’m not sure if this is intended behavior as running the harness with the original tests showed that the harness continued testing even after failing some tests. Comparing to tests written by others, I do not see anything different when compared to mine other than the method of testing I use being more up to date with the latest conventions. I’ve also considered if it is due to synchronous testing, but ruled it out when other tests did not use promises. I am not sure if this is intended behavior or simply a bug.

Step 7: Improving my tests based on inspiration from TypedArray.prototype.reverse

One thing I’ve noticed after observing all the files under test/built-ins/TypedArray/prototype/reverse/reverts.js was the use of a method called compareArray(). While it looks intuitive at first glance, I was curious to find out its inner working since I felt my existing tests could benefit from this function.

Using the “T” shortcut on GitHub, I was able to locate a file named compareArray.js under the harness directory. The function was straight forward and returns a boolean if two arrays are equal in their values. This is exactly the function I need!

I rewrote my tests. This is what it looks like now


Overall, I’ve learned how to get involved in the standardization process of the web, and in particular ECMAScript. I now know how to setup the software necessary for standards testing and how to make use of GitHub’s README.md and CONTRIBUTE.md to contribute to open standards. I’ve also learned more about how JavaScript behaves by doing this exercise. The specification for Array.prototypes.reverse() seemed cryptic at first, but after re-reading the pseudo code a few times, it became clear what its intentions were. It is satisfying to see that all the test you’ve written are passed by Node.JS. I got a glimpse of how great developers build simple functions and unit testing code that is only one to two lines long. It was incredibly easy to understand and pleasant to work with.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s