How to create a private npm.js repository

How to create a private npm.js repository

npmjs.org never goes down

Or so I thought.

I was going to do some development on my local machine, so I fired up npm install <packagename>. Unfortunately, due to a npmjs.org outage, it was not possible for me to get on with my work. So I did what any driven developer would do; I set up a CouchDB replica of npmjs.org. Next time this happens I will be prepared!

As this npmjs.org replica is hosted in the same datacenter as we deploy most of our sites to, it enables a super speedy deployment to testing.

How do we go about this you ask? Well let me tell you.

Installing CouchDB

Note: These instruction are geared towards Ubuntu 12.04 LTS. But it should be pretty easy to get it up and running on *nix systems, including Mac OS X.

Install the required packages:

sudo apt-get install build-essential autoconf automake libtool erlang libicu-dev libmozjs-dev libcurl4-openssl-dev 

Download CouchDB 1.2:

wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org/couchdb/releases/1.2.0/apache-couchdb-1.2.0.tar.gz 

Extract, and relax:

tar xfv apache-couchdb-1.2.0.tar.gz 

Now time to compile:

cd apache-couchdb-1.2.0
./configure
make
make install 

After all that is done we now want to check that everything's fine and dandy, and that we get the expected output:

$ couchdb
Apache CouchDB 1.2.0 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.32.0>] Apache CouchDB has started on http://127.0.0.1:5984/ 

Sweet! But are we responding to requests?

$ curl -X GET http://localhost:5984
 [info] [<0.361.0>] 127.0.0.1 - - GET / 200

Boom, we now have a working CouchDB instance!

You: But wait kind sir, what if we reboot the server? Wouldn't we have to start CouchDB again?

That's correct, we will be adding CouchDB to our init.d scripts, but first we need to create the correct user, group and permissions:

$ sudo adduser --disabled-login --disabled-password --no-create-home couchdb
Adding user `couchdb' ...
Adding new group `couchdb' (1001) ...
Adding new user `couchdb' (1001) with group `couchdb' ...
Not creating home directory `/home/couchdb'.
Changing the user information for couchdb Enter the new value, or press ENTER for the default
  Full Name []: CouchDB Admin 
  Room Number []: 
  Work Phone []: 
  Home Phone []: 
  Other []: 
Is the information correct? [Y/n] Y

User added, now permissions:

sudo chown -R couchdb:couchdb /usr/local/var/{log,lib,run}/couchdb
sudo chown -R couchdb:couchdb /usr/local/etc/couchdb/local.ini

We also want CouchDB to use insecure rewrites for a later step. We can turn this off by editing /usr/local/etc/couchdb/local.ini and adding secure_rewrites = false on line 11 in the httpd section.

$ sudo vim /usr/local/etc/couchdb/local.ini
[httpd]
secure_rewrites = false

Ready, set, link, defaults.

sudo ln -s /usr/local/etc/init.d/couchdb /etc/init.d
sudo update-rc.d couchdb defaults 

Go!

sudo /etc/init.d/couchdb start 

YAY!

Replicating npmjs.org

Now to replicate the npm registry.

curl -X POST http://127.0.0.1:5984/_replicate -d '{"source":"http://isaacs.iriscouch.com/registry/", "target":"registry", "create_target":true}' -H "Content-Type: application/json" 

What this means is that once this is completed your local replication will be an exact copy of the npm registry. However, to ensure we do indeed receive all updates we will add a "continuous":true parameter to the JSON string in our POST request, this utilises CouchDB’s _changes API and will pull any new changes when this API is notified.

curl -X POST http://127.0.0.1:5984/_replicate -d '{"source":"http://isaacs.iriscouch.com/registry/", "target":"registry", "continuous":true, "create_target":true}' -H "Content-Type: application/json" 

We are now replicating continuously from npmjs.org to our private CouchDB instance! If you ever want to stop these replications you can easily do this by running the same command as before but add a "cancel":true parameter to the JSON POST data.

curl -X POST http://127.0.0.1:5984/_replicate -d '{"source":"http://isaacs.iriscouch.com/registry/", "target":"registry", "continuous":true, "create_target":true, "cancel":true}' -H "Content-Type: application/json" 

And we're almost done! All we need to do now is to set up our own version of the npmjs.org registry and we can relax like the humanoid in the CouchDB logo, or as we do @clock, melt into our beanbags...

Getting npm to work with our replicated CouchDB

Most of the steps can be found on isaacs github in the npmjs.org git repositories README

Of course we need to have nodejs and git installed for this:

git clone git://github.com/isaacs/npmjs.org.git
cd npmjs.org
sudo npm install -g couchapp 
npm install couchapp 
npm install semver 
couchapp push registry/app.js http://localhost:5984/registry 
couchapp push www/app.js http://localhost:5984/registry 

Boom, we now have a working npm repository, to test this we can run the following command.

npm --registry http://localhost:5984/registry/_design/scratch/_rewrite login
npm --registry http://localhost:5984/registry/_design/scratch/_rewrite search

If you are getting results then everything has gone according to plan!

So we now have your own privately hosted npm registry, that keeps itself updated. Pretty neat, eh?

All you have to get up and running on your own subdomain is to modify the [vhosts] section in /usr/local/etc/couchdb/local.ini. Uncomment the example and restart CouchDB.

$ vim /usr/local/etc/couchdb/local.ini
[vhosts]
example.com = /registry/_design/scratch/_rewrite

And while we are at it will lock down the application and prevent unauthorised users from deleting our data.

$ vim /usr/local/etc/couchdb/local.ini
[admins]
admin = password
$ sudo /etc/init.d/couchdb restart 

Start using your version of npm with the client!

Straight from the npmjs.org README, just replace <registryurl> with your registries url, for example:

http://localhost:5984/registry/_design/app/_rewrite

You can point the npm client at the registry by putting this in your ~/.npmrc file:

registry = <registryurl>

You can also set the npm registry config property like:

npm config set <registryurl>

Or you can simple override the registry config on each call:

npm --registry <registryurl> install <packagename>

Now you can code and install modules even if npmjs.org is down, or if you want to push the boat even further have this running on your local machine and have an up-to-date npm registry just before your flight. ;]

Thanks to @MikeCronn and @mjorhimself for proof reading this article.

Any questions? Tweet me @tomgco or leave a comment below.

Want to discuss a project?

Got a question? Want some more detail? We'd love to hear from you, contact Jason on +44 (0)1923 261166 or jason.treloar@clock.co.uk.

Related

databreach_istock__matej_moderc_thumb800.jpgRead
Article
26 May 2016

How to stop your customers' data being stolen

If we, as an industry, take anything from the data leaks at TalkTalk, British Gas and Morrisons, it should be that we must take every measure we can to secure customer data. Offering customers a more personalised experience means providing an environment where they are confident that the information they provide will be safe. Collecting and storing customer data and finding out more about your users is key to generating leads and gaining customer insight. But in the rush to get campaigns out the door and find affordable ways to create your digital products, ensuring third parties don’t risk your customers’ privacy and your reputations can be overlooked.

Screen Shot 2016-05-13 at 12.44.37.pngRead
Article
16 May 2016

How to build, test, share and publish a javascript Hybrid mobile application using Cordova

Mobile Applications (Apps) are something every developer wants to create, however, not every developer wants to have to learn multiple languages to be able to create an App which works across different types of devices, such as Android and iOS. Learning Objective C (or Swift) and Java is probably enough to put most people off the idea of creating a cross-platform App. However, it’s possible to create one using technologies which most developers are familiar with. Good old HTML, CSS and JavaScript is all you need. Well, that and Apache Cordova, the mobile application development framework that allows you to build Apps for multiple platforms using a single code base.

Blog-post-img-01.jpgRead
Article
26 April 2016

MongoDB Performance on ZFS and Linux

Here at Clock we love ZFS, and have been running it in production on our Linux file servers for several years. It provides us with numerous excellent features. With the recent release of Ubuntu Xenial 16.04 official support for ZFS is now here, and we are keen to integrate it fully into our next generation hosting stack.

Come and work for Clock

Clock is made up of bright, hard-working and talented people and we're always on the look out for more. You can browse the current jobs below or follow us @clock for the latest vacancies.

View Latest
Vacancies