Fiscal Years & matching data

When you download annual report data from Compustat and need to match this with similar type data from Datastream you need to be sure that when you combine data it is from the same fiscal years. In Compustat you can download the data with both the variables Fiscal Year (= fyear) and Fiscal Year End (= FYR).

In Datastream the database Worldscope provides the annual report data. The data is usually reported with the calendar year as column or row headers. This can be compared to the Fiscal year in Compustat. For the fiscal year end the variable WC05350 can be used.

The unique combination to combine data from these sources is then:

• Compustat Global & Datastream:
  ISIN (or Sedol) + Fiscal year + Fiscal year end
• Compustat North America & Datastream:
  Cusip 9 / Ticker + Fiscal year + Fiscal Year End

N.B.:

1) In Compustat Global you need to download the Currency code as well as the data etc. to know in what currency the data is made available in the database. The data from Compustat North America is reported in US Dollar. In Datastream you can choose to download your data in a specific currency (calculated using historical exchange rates according to Thomson)

2) Accounting Standard can be an important variable as data in databases can be from different statements as reported by a company (statutory reports, SEC filings, etc.). In Compustat this variable is: acctstd. In Datastream it is the variable: WC07536

Email

Compustat database & non-US data

Based on a query from a student I investigated a problem getting output for some variables in the Compustat Global Fundamentals Annual database. I tried to get data for a selection of Dutch and UK companies for the variables: Long-Term Debt – Issuance (DLTIS), and Long-Term Debt – Reduction (DLTR).
No matter what I did, I just could not get data for these variables for the past 10 years.

Many of the regular variables (Total Assets, Revenue, etc.) were not a problem. I studied the manual but this also gave no indication of what I needed to do. Screening variables also did not seem to be causing the problem.

In the end I contacted the the Capital IQ company to ask what the solution was. It turns out that the two variables I was looking for had their origin in filings done with the SEC in America. If I needed data for Dutch or UK companies all I need do is load a list of Global Company Keys (GVKEYs) as a text file in Compustat North America Fundamentals Annual and download the data items there. This was not what I expected but it is good to know the data is available.

N.B.: In Compustat Global you need to download the ISO Currency code as well as the data etc. to know in what currency the data is made available in the database. The data from Compustat North America is reported in US Dollar.

Email

Cleaning Datastream downloads

Using ISIN codes is very often a good choice when you download data from Datastream and want to combine it with other data. ISIN codes are also regularly available in other databases. For the Datastream downloads you will have to seperate the ISIN again from the code of the variable which (as a default) is presented after the ISIN code between brackets.

Because every ISIN code consists of 12 digits, seperating them in Excel is easy: use the function =left(). Example: =left(CELL; 12)

In Stata you can use the substring command substr() to do something similar.
Example: gen isin=substr(Code, 1, 12)

Seperating the code of a company from the code of the variable in Datastream is more difficult if the company codes have varying lengths. The Datastream code (Datastream mnemonic) for instance, is such a code. If you need to seperate this from the code of the variable you need to make use of the length of the variable code which remains the same. To do this we combine to Excel functions: =left() and =len(). Example: =left(Cell; len(Cell)-11)

In Stata a similar thing can be done using a combination of substr() and strlen(). Example:
gen dscd=substr(variable, 1 ,(strlen(variable)-11))

Another thing which you may find necessary is seperating the names of the variables from the names of the companies. The variable name is most easily seperated since the name is a fixed number of characters. Again similar functions using the length of the content of CELLS can be used. A different way of seperating both is using a combination of the Left() function combined with the function Find(). For most downloads from Datastream both items are combined but there is a common element seperating the two, namely: ” – “.
Using this we can build the following formula: =left(CELL;(FIND(” – “;CELL)))

In Stata a similar option is available to split a combined variable. The function split can be used as follows: split Name, generate(deel) parse(” – “). This generates two new variables that both start with the name deel and are numbered, for example: deel1 deel2. Afterwards you rename the variables deel1 and deel2 to what you need.

Email

Excel and Filling up blank cells

Recently I got a question on how to arrange the data in Excel for an analysis where some data was missing. Below here you see an example download from the database Joint Ventures & Alliances from SDC Platinum. Not every row has data but each deal starts on a new row with the date variable “Alliance Date Announced”. Each alliance also has a unique Deal number (3rd column):

In some programs (to do an analysis of the data) what you need is to have the data repeated for each row (observation) related to that specific instance and not any rows with blanks (unless a variable for a row/observation contains no data of course). In the following example the data looks like the previous picture, incomplete:

You want it to look as follows:

Excel has an option that allows you to automatically fill in all the blanks. This works as follows:

Step 1: Make sure that you click on a cell inside the data table

Step 2: In Excel click the option in the Excel ribbon: Find & Select > Go To Special

Step 3: In the pop-up mark the option: Blanks

Step 4: You see that Excel now tries to find all the empty cells in the active table and they are highlighted

Step 5: Use your keyboard to type “=”
Step 6: Now click the Upwards arrow on your keyboard
Step 7: Use the keycombination: CTRL + Enter

N.B. 1: Please remember that after the final row Excel tends to add another row. In this case you need to delete it again as this would only add a duplicate observation to the table.
N.B. 2: Dates may be filled in as numbers. This is fine because in Excel dates are really numbers (number 1 is 1-1-1900). Just use formatting to make sure these numbers are presented as dates again

Warning: When you use the option to complete data for each observation, be sure that the correct data is duplicated. In the example output from SDC Platinum (below) you see columns that relate to events but which are spread over more rows. It may be an idea to copy parts of the data to separate sheets (for instance using unique event/deal numbers) to duplicate data there and later merge the data tables again. See columns G, I and J.

Email

Is it possible to use RIC codes in Datastream?

Recently a student asked the question if RIC codes could be used in other Databases to search for more information on companies. In a nutshell, the answer is: to a certain point and with some manual searches many codes can be indeed turned into codes for other databases if you can use Datastream.

RIC codes are Reuters Instrument codes (Wikipedia). These codes usually originate from Thomson Reuters databases like Thomson One, Eikon or Thomson Reuters News Analytics. The RIC codes for companies usually have a root code and a suffix for the market of a listing. If a company has multiple listings the root stays the same but the suffix gives indicates a different market. A theoretical example: the IBN company has the root code IBM and for a listing at the london stock exchange .L would be added to get the RIC code: IBM.L . If the listing is at the New York Stock exchange .N would be added to create IBM.N .

In the example of the student he got the list from a supervisor and was asked to get some data for these companies. The trouble was: most databases cannot work with RIC codes as these are proprietary and are not included as codes. If your university/organisation does not have a license to Thomson databases (like the ones mentioned above) but does have a Datastream license, the codes can still be converted (somewhat) into codes for this database. Although Datastream is a Thomson Reuters database, and RIC codes (or Thomson1 codes) can be recognised somehow, Datastream sometimes does not provide output or does not return the RIC codes.

Here follows step by step how I still managed to convert the original RIC codes into codes that could be recognised (mostly) by Datastream and return data:

1) Create a new list of codes where the root codes are seperated from the market codes. In excel this can be done by using the function from the Data tab, called: Text to columns. Important to remember here is: not all market codes start with a dot. Some codes stqart with the “equal to” character =. In Excel this character usually signifies the start of a function. Be sure to use the Text to columns option and seperate everything out in one or more rounds and have every column seperated as Text.

2) Create another column with the root codes seperated from the market codes in a similar way as before using the Text to columns option. This time you do not seperate the RIC code out in two text colums. The first column which carries the root code should now be left at “general“:

Some company RIC codes start with numbers that include zero characters in front. These zeros need to stay there because they are part of the code and when converting RIC codes to codes for Datastream these zeros sometimes need to still be there. In essence: we get two columns where you can choose either the original RIC root codes, or the abbreviated ones when necessary.

3) Now create a new column where you recreate the original full suffix code which includes the dot or equal to (=) character. The end result should look as follows:

4) The next step is finding out which RIC market codes represent certain exchanges. I did a few searches and found an excellent site from SIRCA that provided me with a list. I copied and saved this as a text file and imported this text file where each column is a TEXT column. I did not copy the list to Excel immediately from the website as this affected some market codes: specifically the ones that start with the “equal to” character: =
The end result looked as follows:

5) As a next step I matched the market codes from the original list with the list of market codes using 3 times the Excel function =Vlookup(). The result looks as follows:

6) What I now need to do as the final step is create two more columns in case I need to change codes from a specific market/exchange and need to add a suffix code, a prefix code, or both. For the final column I can now use the Excel funstion =Concatenate() to create alternative codes for the RIC codes and have Excel automatically add prefixes or suffixes as required for a list of codes.

7) The last step involves trial and error searches to find out for certain markets what prefix or suffix codes can be added to create either Datastream Mnemonic codes or Local codes that Datastream can actually work with. Example:

Warning 1: Remember that some exchanges that numbers as codes for companies and may require either the original codes with zeroes or not. Choose the relevant Ticker column A or B (which we created) when the Mnemonic or local code requires it.

Warning 2: Most exchanges use common prefixes or suffixes but not all of them: for some exchanges you may need to look up all codes!

8) Using the list or newly created codes we can now do a Static search in Datastream to find out if all the codes are stil current and which are not: the original RIC code list may contain codes that have changed and some codes were deleted (even though the company data is still there). First copy the new codes to a new sheet (or column) and paste them as values to get rid of the Excel formulas. Then upload them using the Datastream option: “Create List(From Range)“:

9) Next Do a Static Request search and use the list to get some basic data back that signifies which companies are associated with these codes. In the example search I requested the following codes back: DSCD (Datastream code), NAME, ISIN code. The list of codes that we uploaded at step 8 we can find/use by clicking the List Picker button below the orange Find series button.

If the original list of RIC codes is a recent one, chances are you will get many good codes back which can then be used to get actual data from Datastream or another database. The result looks something like this:

N.B.: When you have access to databases like Amadeus, Orbis, Orbis Europe or Compustat you can also try uploading the RIC root company codes as tickers and see whether they are recognized there as well. This method is more quick & dirty (tricky), as you do not know whether the right listings and ticker codes will be available for the companies.

Email

Excel and duplicates in a dataset

In the past I posted items on duplicate data from datasets (from Compustat databases) and how to find this out using programs like stata. For smaller datasets the program Microsoft Excel can also be used to investigate your datasets. AT some point, when merging datasets from multiple sources it may happen that you get duplicate data. Using Excel functions like a pivot table (draaitabel) and vlookup (vertikaal zoeken) duplicate data can be detected as follows:

1) first you need to tell Excel that your dataset is a specific delineated table using the menu option: Insert > Table

2) When you insert this option you need to indicate the range and if there are any column headers

3) The same menu tab insert (invoegen) has the option to create a summary pivot table (samenvatten met draaitabel). Select this option at the top left corner:

4) Create the PivotTable in a new (or empty) sheet:

5) The empty Pivot Table will be shown as follows:

6) In this example I drag the fields I want to check for duplicates (records/observaions) down to rows (Rijen). At the box values I drag a random field (in this case indfmt). On the left side of the screen the result will be presented:

7) In this example I click the field gvkey and choose the option to change the field settings

8) At subtotals & filters mark the second option No(ne)

9) In the tab Format & Print (Indeling & Afdrukken) mark the option: item labels in table format (Itemlabels in tabelvorm weergeven). Also make sure to mark the option to repeat item labels (Itemlabels herhalen):

10) The result chould look as follows:

11) Copy the list to a new sheet and use the concatenate function (Tekst.samenvoegen) to create a combination of GVKEY and year:

12) Create a similar link list with the Concatenate function in the original Compustat datasheet

13) Now use the VLOOKUP function in Excel in the Compustat datasheet to look up the link in the sheet with the result of the Pivottable. Make sure that the VLOOKUP option says False (Onwaar) at the 4th option as follows:

14) Finally, using the filter option you can find out if there are duplicatesd by selecting everything for which the count is higher than 1. The tricky thing will be deciding what to do with the result.

N.B.: If a dataset has over 100.000 observations the process described above will take some time as Microsoft Excel will require significant processing power from the computer. For larger datasets I reccomend using Stata.

Email

Using Stata to count a sequence

Not long ago a student asked me how to calculate the tenure for auditing firms that were attached to firms within a certain time frame for up to a specific year. The student was interested in the effect of auditor tenure on companies. The subject of auditor rotation has become interesting as a research subject in the field of accountancy as in some countries firms are required to change their Auditor every few years to ensure auditor independence.

Using the Stata program this can be done using a script to identify “spells”. My colleague Matthijs de Zwaan helped me with this and created an example script based on an article in the Stata Journal.
The script I made was based on the example and can be adapted to count other kinds of sequences in datasets.  I also added a few lines of code to show how a dummy could be created to identify if the auditor in the example Excel file was a Big Four accounting firm or not. The script can be downloaded here and looks as follows:

The end result (after removing suplicate data) gives the tenure of the last auditor (latenure) for each firm at the year 2008:

N.B.: In the .do file the location for all files is the I: drive. You may need to change the drive letter in the original script to (for instance) c: or H: to get it to run. Make sure both the script file and Excel dataset are in the same location.

Email