Tuesday, April 3, 2012

unofficial user guide for ARRL’s Logbook of The World

unofficial user guide for ARRL’s Logbook of The World

I’ve created this unofficial user guide for ARRL’s Logbook of The World to supplement the help available from the LoTW site. It is an incomplete work-in-progress. Feedback and additional content is very welcome.

Quick links
Why should I bother with LoTW?
Joining LoTW
Using LoTW - some Hinson tips
Checking/correcting your CQ and ITU zones and locator in LoTW
Applying for DXCC or WAS awards using LoTW
Archive your LoTW certificate/s and log/s [important - trust me!]
Generate an ADIF of your DXCC credited (granted) QSOs
Find out who else is using LoTW
“Official” help on LoTW
A wishlist for LoTW enhancements

Why should I bother with Logbook of The World (LoTW)?

If you are a serious DXer, you know how frustrating it can be waiting for QSLs to arrive to confirm each new country. Whether you send QSLs and $ direct or just hope for cards to arrive via the worldwide QSL bureaux, collecting confirmations on paper can be tedious and expensive. Then, once you’ve got your little stack of QSL cards and you wish to apply for awards such as DXCC or WAS, you still need to get them checked and verified by the relevant organization.

Quite simply, LoTW makes the whole process quicker, easier, cheaper and safer.

With LoTW, you upload your log periodically to the ARRL’s LoTW website where it is entered into a database system that cross-matches your QSO details against other uploaded logs, generating electronic confirmations for all QSOs that match. LoTW confirmations of matching QSOs normally come through within days or weeks, as soon as both parties to a QSO upload their logs to LoTW. The LoTW system can also track your progress on DXCC and WAS awards (only those two at present, but CQ Magazine’s awards will soon be added) and, when you are ready to claim your awards, it handles the application paperwork electronically for you. For QSOs that have been verified in LoTW, there is no need to sort out QSL cards, fill out the application form and submit the cards for verification. However you may still choose to submit QSL cards for specific DXCC countries or US states for those bands and modes which are not yet confirmed on LoTW (i.e a “mixed” DXCC application) ... or you can just be patient in the hope that eventually you will get them all in LoTW anyway.

By the way, there is absolutely nothing stopping you exchanging paper QSLs as well as using LoTW. Many of us enjoy collecting those picture postcards from exotic foreign places to show off to our ham friends and amaze our families. Even if you are not collecting QSL cards, I encourage you to respond to incoming paper QSL requests as a courtesy.

Back to quick links

Joining LoTW

The process to join LoTW is a little complicated because it is a secure system. It uses digital certificates to authenticate submitted logs and prevent people uploading false or fraudulent logs for others (authenticated users are trusted to submit accurate logs for themselves!). So the first stage is to obtain your digital certificate from ARRL.

Obtain your digital certificate from ARRL

First you need to download and install the TrustedQSL (tQSL) programs from here. There are two associated programs: tQSL.exe is the program you will use to digitally sign your logs, andtQSLcert.exe is the program used to manage (including apply for) your digital certificates.

Using tQSLcert, you now generate an electronic digital certificate request which is saved as a .tq5 file. This involves entering basic information such as your name and callsign and creating a file that you will submit to ARRL. They then check the details. US hams’ details are verified using the FCC’s systems. Non-US hams have to submit a copy of their license by post to get it checked against the certificate request. If everything is OK, ARRL sends you your digital certificate.

Now you load the digital certificate onto your system using tQSLcert again. The certificate is checked against the certificate request and electronically verified, just in case anything got corrupted on the way. If it all checks out, you can now start using the digital certificate to sign your logs and upload them to LoTW.
Upload your first log to LoTW

These are the steps involved in uploading your log to LoTW:
Generate an ADIF output file from your logging program containing the QSOs you want to upload to LoTW (i.e. your whole log at the start. Subsequently, you will normally only send new QSOs since the previous LoTW upload);

Electronically sign the ADIF file using tQSL . This generates a digital signature based on the content of the ADIF file and your private key, allowing ARRL (or indeed others) to verify the data using your public key on your digital certificate. The digital signature is appended to the QSO data in the ADIF file and saved as a new .tq8 file;

Login to LoTW on the ARRL website;

Click the Upload tab in LoTW;

Find and select the .tq8 file you just created and wait for it to upload. This may take a few minutes if you have thousands of QSOs to upload but is nearly instantaneous if you are only uploading a few;

Check for new LoTW QSLs using LoTW’s reporting tabs for QSOs or awards;

[Optional step but a worthwhile habit to form] Backup your log!

Back to quick links

Using LoTW - some Hinson tips

Upload your log to LoTW as often as you can , ideally at least once a month, preferably once a week. If you are as active as I am and working hard to improve your DXCC scores, you may like to upload your log daily and check your LoTW QSLs at the same time.

Keep an eye on the QSO and QSL counts (top right of most LoTW screens). As LoTW usage spreads, the proportion of QSOs that are QSLd via LoTW is gradually increasing. Currently about one third of my QSOs are confirmed via LoTW.

If you alter any logged QSOs (for example correcting broken callsigns when QSL cards arrive), you need to re-upload the changed QSO records to LoTW . While you might be able to extract the changed QSOs and just upload them, the easiest way is to periodically re-upload your entire log since LoTW automatically ignores exact duplicates and only saves any changes . However, please don’t do this too often (no more than, say, once or twice a year) as it wastes computer power and slows the LoTW systems down a bit. Be nice.

Use the award status table to check on your DXCC or WAS status, and the quick QSL report to check which QSOs have been confirmed lately.

Work lots of contests ! Contest stations are more likely to upload their logs to LoTW than most hams. Most big DXpeditions also use LoTW, although there is a tendency to wait several months before uploading in order to encourage people to send a few extra $$ with their direct QSL requests :-(

Preferentially contact other LoTW users rather than non-users, given the choice. The LoTW user list is helpful to identify them, especially if you integrate the data in your DXcluster or logbook software (VE7CC’s cluster user program is ideal for this). This is not about being a clique, simply that you are more likely to get a QSL and do so more quickly via LoTW than by any other means.

If you are patiently awaiting a LoTW QSL that seems “overdue”, the DX may not have uploaded his log in a while (if ever!) or it may have been a busted QSO. You can check the last time someone uploaded their logs to LoTW on this page.

You may need the security of a password to unlock access to the secret key on your LoTW certificate, but then again you may not. If the risk of some reprobate sneaking onto your machine and maliciously signing false logs in your name is low enough to be safely discounted, then feel free to remove the passwords from your certificates in tQSL using theinstructions here. If you do this, you can upload log updates to LoTW from Logger32 with “just” 5 clicks following the instructions on my Logger32 page using a “utility” entry to call tQSL with the appropriate parameters.

If you need to download your entire log from LoTW as an emergency backup*, there are at least five options:
Use LoTW’s QSO reports to find out all you can about the missing QSOs and re-enter them manually into your logging program (tedious and error prone unless you are only missing a few QSOs);
Capture (‘screen-scrape’) the text one-screen-at-a-time from LoTW’s reports: this is also tedious and error prone, and is definitely not recommended;
Use the neat online LoTW log download utility by K1MU, without entering any specific QSO criteria;
Use this LoTW query page directly (it starts a full log download - thanks to G4LMW for the tip);
Use the ‘synchronise log with LoTW’ function in logging programs such as AClog (but not yet Logger32, unfortunately).

* Treat this as a last resort: LoTW only stores the basic, minimal QSO details, approximately 2 Mb of ADIF file per 10,000 QSOs. It is MUCH better to make your own regular off-line log backups, saving your ADIF file to, say, a CD or USB memory stick from time to time.

Back to quick links

Checking/correcting your CQ and ITU zone and locator in LoTW

It seems some hams are either in such a rush or so confused when they join and start using LoTW that they either make leeetle mistakes with their zones and Maidenhead locators, or leave them blank. If the values are wrong, every LoTW update thereafter and hence any LoTW QSLs created carry the wrong info until eventually someone notices and kindly lets them know.

With ARRL planning to adap LoTW for the CQ awards, it is going to be interesting to see how the many CQ zone errors will be addressed.

Anyway, here’s the process to check and if necessary correct the data:
First create and save an ADIF version of your log containing all QSOs from the location whose zones/locator you want to correct. [In Logger32, this means going to "File" then "Export Logs" then "Create ADIF (Adi) file". Navigate to a suitable directory and give the ADIF file a name. Do not select "Partial log" as you almost certainly want the whole thing. I normally select "Export full Country names" here as it lets me check and sometimes correct the DXCC country allocations made by Logger32].

Make offline backups of your log at this point, not because anything in the rest of this procedure is particularly risky, but just in case. It is generally safest to make and store copies of both the ADIF log (which may contain errors from the export process) and your logging program’s proprietary format (which may contain additional info but is not so portable as the ADIF). Be sure to store them safely offline on suitable storage media such as CD or USB memory stick.

Run tQSL.exe

On the top line menu select "File" then "Sign existing ADIF or Cabrillo file".

Click once on a "location" to select it [note: here you are actually selecting the appropriate digital certificate - if you manage more than one callsign in LoTW, each will have its own "location" defined and you must use the appropriate ones to sign the relevant logs].

Click the "Edit" button.

Check and if necessary correct the information there, provided you actually know the correct values! If not, look up your CQ zone, your ITU zone and your Maidenhead locator. [Some LoTW users claim that, although they didn’t enter anything, random values appeared here. I’m not convinced. Anyway, please take the time to find and enter the correct values for your QTH , as this info will be sent via LoTW to all the people your QSO and is a pain to correct later.]

HINT: if your ITU zone number is less than your CQ zone numberbut you don’t live in Canada, you have probably made a mistake. Even if you do live in Canada, double-check them anyway!

A blank or null entry doesn’t overwrite whatever value the other people you have contacted might have assumed and logged. A wrong entry may overwrite their correct data and is unhelpful.

Click "Next".

Click "Finish".

[The "location" you just corrected should still be selected] Click "OK".

Find the ADIF file that you created in step 1 and click "Open".

Click "Save" [the signed log file will be saved with the same file name as the ADIF file but with a .tq8 extension, in the same directory as the ADIF file by default].

Click "Yes" to confirm that you really want to sign your log [doh! Why else would you be doing this? Dumb question!].

When prompted, enter your tQSL password [to unlock your private key and sign your log with your digital certificate] and click "OK" to sign your log.

Login to the LoTW website and upload your signed log as usual. LoTW overwrites your previously-uploaded data with the correct zones etc., unless any QSOs have already been QSL’d in which case those records are locked.

[Optional] If you have more than one location and log to sign and upload, go back to step 1 and run through again for the other info.

Thanks to Chris for helping prepare these instructions, and for correcting his info!

PS Version 1.13 of tQSL allegedly performs some checks on the location information. If this is true, the data in LoTW should gradually improve over time as hams have to apply for replacement certificates every few years, and the process forces them to upgrade to v1.13 or later versions to load the new certificate. Meanwhile, the location information is plain wrong for a significant percentage of logs in LoTW.

Back to quick links

Applying for DXCC or WAS awards via LoTW

The LoTW system can automatically track your progress towards ARRL’s DX Century Club and Worked All States awards but first you need to configure it by setting up your DXCC and/or WAS accounts in LoTW.

The DXCC rules allow you to accumulate QSLs from more than one personal callsign (for example if you use a personal contest or vanity call but not a shared/club call), provided all QSOs are made from the same DXCC country. Under WAS rules, all QSOs must be made from within 50 miles of the same location: if you move more than 50 miles away, you have to re-start your WAS claims from the new QTH. Configuring LoTW therefore involves telling the system where you operate from.

When LoTW’s DXCC or WAS reports indicate that you have enough QSLs for an award, you can prepare and submit your application online, sending the fee to claim the award.

At ARRL HQ, the awards process through LoTW is much less labour-intensive and quicker than manually checking written applications and QSL cards.

If you have additional paper QSLs to submit, you can send them to ARRL HQ (necessary for all 160m cards), take them to a ham convention if card-checkers will be present, or send them to your nearest DXCC field checker (in my case that’s the Oceania field checker Lee, ZL2AL). In the DXCC office at HQ, the verified DXCC confirmations from QSL cards are simply added to those confirmed and claimed via LoTW and appear in your LoTW DXCC records. [Note: this process takes a while - 4 months in my case because I applied for the award at the end of the calendar year when many honor rollers are doing their annual updates]. Be patient and get in touch with the DXCC office if the process seems to have stalled for more than, say, 2 or 3 months.]

Back to quick links

Archive your LoTW certificate/s

The “.p12” (saved certificate) files are particularly important: if you somehow lose or corrupt your certificates (for example if your hard drive crashes, you pick up a PC virus, you accidentally delete or overwrite the files, or your shack burns down) and have no archive, you will need to go through the rigmarole of reapplying to ARRL HQ, sending your license again for validation.

If you own or manage several calls in LoTW, you should create and archive the .p12 files for each call. Do it now, before you forget!

How to archive your LoTW certificates:
Run TQSLCert.
Select a certificate (you may have more than one if you own or manage several calls).
From the menu, select Certificate then Save.
Navigate to a convenient directory on your system.
Click Save.
The software prompts you for a password to secure the saved certificate. Enter and confirm a password if you wish (provided you are certain you will be able to recall it much later if you need to recover your certificate!), or just leave both fields blank if you don’t want to set a password.
The software will prompt you for the existing password to open the certificate, if you have previously set one.
Click OK. The software confirms that the certificate has been saved as a .p12 file. Click OK again.
If you have other certificates, go back to step 2 and save them all one at a time in the same way, in the same directory. Be sure to get them all.
Exit from TQSLcert.
Copy all your .p12 files to a suitable archival/long-term storage medium such as (in decreasing order of merit) CD-ROM, DVD-ROM, external hard drive, USB memory stick, Web or cloud backup, another PC, 8” floppy diskette, stone tablet, wax disk, sand impression.
Store your archive safely, well away from your computer and shack (don’t just sling it in the bottom of a drawer: it’s valuable!).

To recover from a lost certificate, run TQSLCert, select File from the menu then Load Certificate File, then find the .p12 file you saved safely away for a rainy day and enter the password if you set one. Easy peasy, provided you still have that .p12 file and remember the password!

Backup your log/s too

Be sure to take regular offline backups of your log or logs . Without backups, if your log is damaged or lost, you will lose all record of all the QSOs you’ve ever had, except perhaps for the very basic details of any you have previously uploaded to LoTW or Clublog (both of which only store the bare minimum of QSO information - think of them as last resort backups). Believe me, it’s a real pain when it happens.

Those who worked Mellish Reef toward the end of a certain DXpedition know what I mean: someone accidentally dropped the laptop into the sea while leaving the reef. Ooops. Luckily QSOs made up to a day or so earlier had been uploaded to a control station and were safe but the final QSOs were lost forever to Neptune. Doh!

Take backups of your log and save them to suitable offline media (see above). Make this a regular routine e.g. once a month or every quarter depending on how active you are - once every few hundred QSOs is another way. At the very least, take a moment to archive your entire log to date every Christmas/new year, and store it safely away. That way, the most you will lose will be all the QSO’s you made during the subsequent year - still bad news but not a total disaster!

Note: these offline backups should be in addition to more frequent online ‘convenience’ backups, where you simply save a copy of your log file to another directory on the C: drive or another hard drive on your machine. Online backups are handy to recover from little accidents: provide the online backup file is OK, you can recover most of your QSOs (up to the last backup) after accidentally corrupting a section of your log, or perhaps deleting or overwriting the whole log file.

Back to quick links
Generate an ADIF of your DXCC credits (granted)

The QSOs that have been confirmed and credited to your DXCC record can be downloaded from LoTW as an ADIF file in order to update your local DXCC records (e.g. in Logger32) automatically, without the need to find and update the status of every credited QSO individually.

Unfortunately, LoTW doesn’t offer a specific report for this so the trick is to use this wonderful little freeware utility by SP7DQR (thanks Marek, what a star!).

Marek’s DXCC credit downloader utility asks for your LoTW username and password, then logs in to LoTW. It generates working lists for all your DXCC records (e.g. DXCC Mixed, DXCC CW etc.) and then one-by-one looks up, finds and downloads the relevant QSO info from LoTW, saving it as an ADIF file. It is a slow process, taking about 4 seconds per QSO (so for my 1500 credits, the little blue progress bar took about an hour and 40 mins to reach the end stop) but the utility is perfectly happy to churn away in the background whereas I would have gone nuts trying to do what it is doing by hand!

The next step is to load the DXCC credits into your logging program. Marek’s utility includes the function to take in your complete ADIF log file, update the DXCC credit status for the relevant QSOs, and output an ADIF log file containing all your credits. This works much quicker, taking just a few seconds in my case ...

The utility adds the credit information to the ADIF using two fields. For example, the ADIF record for my 10m QSO with TF3Y goes from:

<BAND:3>10M <CALL:4>TF3Y <CONT:2>EU <CONTEST_ID:8>ARRL 10m <CQZ:2>40 <DXCC:3>242 <FREQ:9>28.094200 <GRIDSQUARE:6>HP94bd <IOTA:6>EU-021 <ITUZ:2>17 <MODE:2>CW <OPERATOR:4>ZM4G <PFX:3>TF3 <QSO_DATE:8:D>20111210 <RST_RCVD:3>599 <RST_SENT:3>599 <SRX:3>528 <STX:3>665 <TIME_ON:6>211616 <TIME_OFF:6>211616 <LOTW_QSL_SENT:1>Y <LOTW_QSL_RCVD:1>Y <APP_LOGGER32_QSO_NUMBER:5>31578 <FREQ_RX:9>28 .094200 <EOR>

... to ...


The text in bold has been prepended to the record. The utility adds both <CREDIT_GRANTED> and<APP_LOGGER32_CREDIT_GRANTED> fields because Logger32 uses the additional information about whether the QSO was credited as a result of a QSL card or a LoTW match.

The utility does some integrity checking along the way, generating a plain text error file with information on any credited QSOs it didn’t find in your log - nine QSOs in my case:

HS0ZEA, for example, is the call credited in my DXCC records but in fact he was mobile, and I logged him as HS0ZEA/M (which is the callsign he used on air). LoTW evidently strips off the /M suffix, giving a mismatch with the QSO in my log. It is easy to fix these nine errors: I will simply have to edit my log manually to show that these QSOs have been credited for DXCC.

First, though, I must import the ADIF file containing the credits into Logger32: there may be a function in Logger32 to import and synchronize the amended ADIF into my log, but I prefer to create a new logbook and import the amended ADIF there.

So far, Logger32’s the DXCC report has gone from looking like this:

with a number of pink G (= Granted) blobs that I had entered previously ... to this:

with extra pink G blobs as expected. However, I noticed that one pink G blob (right hand blob on the second row) has now reverted to a plan orange C (Confirmed) blob. How odd!

Digging a little deeper, I find that it is a QSO with 4L Georgia on 40m, specifically 4L2M. He has confirmed the QSO on LoTW and it has been credited to my DXCC record:

The QSO is correctly listed in the ADIF file containing the credits as follows:


It appears Logger32 failed to register the credit when importing that QSO for some unknown reason. Could be a bug in the statistics function (there are others!). Anyway, the usual stats recalc fixed it:

So that’s it. A way to update your DXCC records to GRANTED status using LoTW and Logger32.

Back to quick links

LoTW users

Robert HB9BZA maintains a handy list of LoTW users compiled from “lotwreport.adi” files submitted by those of us who like to help. The cluster user program by VE7CC uses the list, adding a “+” to the comments field on spots for DX stations who use LoTW. If two or more wanted DX stations are spotted, I give preference to those with the plusses because I’m more confident of getting a QSL from them.

ARRL, unfortunately, does not release this information directly so it relies upon those lotwreport files to identify who is QSLing via LotW. If you use LoTW, please send your lotwreport.adi files to HB9BZA every so often (maybe every 500 to 1,000 LoTW QSLs) and help maintain the list.

If you send your lotwreport to Robert, he kindly responds by emailing you back with a list of any new callsigns he’s added to the list.

Back to quick links

Getting help on Logbook of The World (LoTW)

The web page you are reading is “unofficial”, just a compilation of things I have found out while using LoTW.

Before you log-in to the LoTW system, there is a help and FAQ file by ARRL - not exactly the most helpful source of information.

If you have other queries not answered in this FAQ or the ARRL files, you could try asking me but you are likely to get a more accurate response from the nice people at ARRL HQ (e.g. try Bill Moore NC1L for DXCC or WAS queries), or by joining the LoTW email reflector at Yahoo!.

Back to quick links

A LoTW wishlist

Here are some changes to LoTW I’d quite like ARRL to make:
Anything . What I mean is I’d like ARRL to update LoTW at all. It seems to be cast in stone at present.

Better online help for users who have logged-in to LoTW. The current online help is useful but is only available from the main LoTW page before you login to LoTW. Logged-in users are left to flounder around helplessly, as it were. [ARRL is planning to redevelop the LoTW site and have promised more help.]

More awards to claim , in addition to DXCC and WAS. IOTA, for example, would be good and there are many many more worldwide. ARRL would do us all a service by opening up the system to integrate other awards, not least because it would encourage more people to join and use LoTW.
Alternatively, ARRL, how about modifying LoTW to generate verified and signed logs that can be submitted to other organizations for their awards? LoTW could generate and append its own digital signatures on its output which other organizations could then verify before accepting them.

A significant reduction in the award application fee for awards that are verified electronically. Since QSOs are cross-checked automatically by the system, the amount of manual checking is minimized, so we ought to pay much less. This would encourage yet more people to use LoTW. [I believe some of the DXCC fees are already reduced or waived for LoTW updates.]

More marketing/publicity and proactive support for LoTW from ARRL and ideally other ham organizations. When enough hams are using LoTW, it will reach an “event horizon” where the benefits of using it encourage practically all active hams to join up. We’re getting closer but not quite there yet ...

More feedback on the status of award applications. It would reduce the stress if we knew that our DXCC or WAS applications were progressing e.g. moving slowly up the queue, awaiting card confirmations, awaiting final checks or whatever. If the checking process is largely automated, the status updates could presumably be generated automagically too...

Please contact me if there are other ideas you would like to add to the list, or to comment on my suggestions.

Notwithstanding my little wish-list, I’d like to thank ARRL and the DXCC desk for making this facility available to hams worldwide. I appreciate that a PKI system of this scale must have involved a substantial investment and incurs ongoing maintenance and operational costs. Thanks!

No comments: