.jpeg){kind=avatar}
Writing a node js package manager with Typescript
This article is not necessarily a code walk through, but rather my experience rewriting a package manager I once wrote in Rust here to Typescript code here.
Before we dive into the fun stuffs, let us first of all understand what a package manager is. As programming languages grow and are used to build more complex applications, it makes sense to have reusable code (libraries) across code bases for easier and faster development. These libraries have to be managed by some form of managers so that new incoming devs would have a way to download and use them from an already specified list. Example of popular package managers include npm and yarn for JavaScript, pip for python, Gem for Ruby, etc.
In this article, we would try to mimic some of the implementation found in Yarn and NPM to manage our JavaScript libraries.
Note: I decided to use a
pkg.toml and pkg.lock.toml file instead of the traditional package.json file we are alll familiar with. I also ignored extra symbols like ~ < ≤ > * in the package version number to keep it simple.
Let's dive in.
This article would be divided into multiple sections, where each section attempts to solve a particular problem(s).
1. Reading input from
CLI and TOML file
2. Getting a map of the latest dependency version from CLI input
3. Getting all related dependency that maps to a particular package
4. Downloading and extracting the compressed folder into it’s appropriate directory
5. Updating the package in the TOML and lock TOML file
6. Running commands specific commands from the TOML file akin to npm run <command>
7. deleting package(s)
Reading Input from CLI and TOML file
The first step to building a
CLI, is to make the index.ts file run as a binary using a shebang. What this does is it makes our TS file run as an executable, so instead of running a command like ts-node index.ts we could just run ./index.ts. You can read more about shebang here.
In our case we can easily add this to the top of the entry file:
#!/usr/bin/env ts-nodeThen we need to
chmod the file and just like that we can run our file just as we would run an executable.
Based on how
npm works, we can pass packages in the package.json file to be installed or via the CLI. Our CLI also has to give users the option to do the same but in this case a pkg.toml file. The way we decide where to install from is quite simple. If no argument is supplied after the install or i command, then we install all packages specified in the toml file else we install the packages specified via the CLI.
To read args from the
cli input, we simply use the process.argv provided by JS and splice out inputs we don’t need, while to parse TOML files we use a TOML parsing library https://www.npmjs.com/package/@iarna/toml
import { JsonMap, parse } from "@iarna/toml";
export const parseToml = (payload: string): JsonMap => {
const obj = parse(payload);
return obj;
};import * as fs from "fs";
export const readFile = (path: string): Promise<string> => {
return new Promise((res, rej) => {
fs.readFile(path, "utf8", (err, data) => {
if (err) rej(err);
res(data);
});
});
};//read and parse toml file
const toml_data = await readFile("./pkg.toml");
const obj = parseToml(toml_data);
//if arg is passed, read arg from cmd else read toml file
if (process.argv[2] === "install" || process.argv[2] === "i") {
//toml config
if (process.argv.length == 3) {
//get dependency list
const dependency_list = obj["dependencies"] as object;
//get dev dependencies
const dev_dependency_list = obj["devDependncies"] as object;
//merge both dep and devDep
const all_dependencies = { ...dependency_list, ...dev_dependency_list };
} else {
//read arg from command line
//get the last n elem in the array, removing the first 2
const dependecy_list = process.argv.slice(3);
}
That seemed pretty straightforward 🙂
Getting a map of the latest dependency version from CLI input
For packages listed in our
TOML config, we would easily use the version specified there eg
[dependencies]
fs-extra = "10.1.0"
core = "1.0.113"But for packages supplied via the
CLI, our package manager currently has no way to specify the version you need to install via CLI. So we would just install the latest stable version for every package installed via this route.Thankfully,
NPM has a publicly accessible API that we could use to get the latest version of a packageconst base_url = "https://registry.npmjs.org";const getLatestVersion = async (
dependecy: string
): Promise<string[]> => {
try {
const { data } = await axios.get(`${base_url}/${dependecy}`);
return [dependecy, data["dist-tags"]["latest"]];
} catch (err) {
throw err;
}
};We can then update the section of our entry point that reads arg from
CLIif (process.argv[2] === "install" || process.argv[2] === "i") {
//toml config
if (process.argv.length == 3) {
...
}else{
//craete an empty JSON map
let cmd_map: JsonMap = {};
//read arg from command line
//get the last n elem in the array, removing the first 2
const dependecy_list = process.argv.slice(3);
const latest_list = [];
for (let i of dependecy_list) {
latest_list.push(getLatestVersion(i));
}
const resolved_list = await Promise.all(latest_list);
for (let item of resolved_list) {
cmd_map[item[0]] = item[1];
}
}I decided to store the dependency and version number in a hash map as it would be used later.
Now we are able to read input from the
CLI and TOML , store dependencies from TOML in a JsonMap, get the latest version of CLI packages and also store in a JsonMap. Our package manager is taking shape 😃
Getting all related dependency that maps to a particular package
Now that we have all our dependencies and version in a
JsonMap, we still need to do one more in terms of getting dependencies and versions.Let us imagine we want to install
express.js, how NPM works under the hood is to fetch all immediate dependencies of express and the dependencies of those dependencies and so onEg
express depends on qs which depends on side-channel and that might also depend on something else.We need to find a way to handle these immediate dependencies and all nested sub-dependencies up to the
nth depth using recursion. I decided to follow a different approach though, get all immediate dependencies of a package, store them in a hash map along with their version, iterate the hash map and recursively get all sub-dependencies to the nth depth.Let’s see how that looks
const getImmedteDep = async (
dependecy: string,
version: string
): Promise<{}> => {
try {
const { data } = await axios.get(
`${base_url}/${dependecy}/${await getVersion(dependecy, version)}`
);
//get immediate dependencies of the dependecy
const related_dep = data["dependencies"];
return related_dep;
} catch (err) {
throw err;
}
};const getNestedDep = async (
dependecy: string,
version: string
): Promise<{}> => {
try {
const { data } = await axios.get(
`${base_url}/${dependecy}/${await getVersion(dependecy, version)}`
);
let all_dependecies = {};
const related_dep = data["dependencies"];
const all_related_dep = { ...related_dep };
//if there are nested dependecies, recursively call the function
if (Object.values(all_related_dep).length > 0) {
//append to the dep map
for (let item in all_related_dep) {
all_dependecies[item] = all_related_dep[item];
let v= await getNestedDep(item, all_related_dep[item] as string)
for(let a in v){
all_dependecies[a] = v[a];
}
}
}
return all_dependecies;
} catch (err) {
console.error(err);
}
};Remember that we want to ignore all special char from the version number for simplicity:
getVersion = async (
dependency: string,
version: string
): Promise<s;string> => {
if (version.includes("*")) {
const ver = await getLatestVersion(dependency);
return ver[1];
} else {
return version
.replaceAll("~", "")
.replaceAll("^", "")
.replaceAll("*", "")
.replaceAll(">", "")
.replaceAll("=", "")
.replaceAll("^", "")
.replaceAll("<", "")
.trim();
}
};That might be a lot to take in, so you can take a pause and look through the code again
Downloading and extracting the compressed folder into its appropriate directory
Now that we have a huge hash map of all the immediate and nested dependencies, we can go ahead to download the compressed package, extract them and move them to where they belong.
The first step is to get the link of all the tarball files and the dependency name.
const getTarballLinkAndName = async (
dependecy: string,
version: string
) => {
try {
console.log(`downloading ${dependecy} dependencies...`);
const { data } = await axios.get(
`${base_url}/${dependecy}/${await getVersion(dependecy, version)}`
);
const download_link = data["dist"]["tarball"];
const name = data["name"];
...
} catch (err) {
console.error(err);
}
};Now, we should download the file into the
node_modules directory and extract the file. But when the file is extracted, we notice one funny thing about compressed NPM packages, they all live in the package folder. Eg the express library would be found in express/package/stuff that needs to be changed to express/stuff else node would not be able to use the packageconst dir_name = "node_modules";const getFileName = (path: string): string => {
var parsed = urlParser(path);
return basename(parsed.pathname);
};const downloadAndunZip = async (link: string[]) => {
try {
console.log(`Extracting ${link[1]}`);
if (!fs.existsSync(`./${dir_name}`)) {
fs.mkdirSync(`${dir_name}`);
}
const fileName = getFileName(link[0]);
const module_location = `./${dir_name}/${fileName}`;
const file = fs.createWriteStream(module_location);
const request = https.get(link[0], function (response) {
response.pipe(file);
file.on("finish", async () => {
//extract zip and copy to right location
file.close();
await unZip(module_location, `./${dir_name}/${link[1]}`);
});
});
} catch (err) {
console.log(err);
}
};As seen above, we need to download the file to the filename path which is usually in the format
express-1.2.3.tgz, extract that into the express folder, delete the tarball, and move all files from the package folder outside.const unZip = async (module_location: string, newWrite: string) => {
decompress(module_location, newWrite)
.then((_files) => {
//delete the tarball
removeSync(module_location);
moveFolder(`${newWrite}/package`, newWrite);
})
.catch((err) => {
if (err) console.log(err);
});
};const moveFolder = (src: string, des: string): void => {
try {
copySync(src, des);
//delete everything on the package folder
removeSync(src);
} catch (e) {
console.log("error enconutered");
}
};
Updating the package in the TOML and lock TOML file
Now that we have the download and extract completed, we need to be able to update the
pkg.toml file to reflect dependencies installed via the CLI, and create a pkg.lock.toml file. But for our case, we would keep this file pretty simple. It would simply be a map of all the related dependencies.First, let’s create an
install function that could be used across many installation optionsconst install = async (depenecyMap: JsonMap) => {
let dependecy_graph: JsonMap = {};
for (let cli_dep in depenecyMap) {
console.log("getting dependency list...");
dependecy_graph[cli_dep] = {};
dependecy_graph[cli_dep][cli_dep] = depenecyMap[cli_dep];
const immediteDep = await getImmedteDep(
cli_dep,
depenecyMap[cli_dep] as string
);
for (let immed_dep in immediteDep) {
dependecy_graph[cli_dep][immed_dep] = immediteDep[immed_dep];
const nestedDep = await getNestedDep(
immed_dep,
immediteDep[immed_dep] as string
);
for (let nested_dep in nestedDep) {
dependecy_graph[cli_dep][nested_dep] = nestedDep[nested_dep];
}
}
}
const download_list = [];
for (let j in dependecy_graph) {
let map_obj = dependecy_graph[j] as JsonMap;
for (let i in map_obj) {
download_list.push(getTarballLinkAndName(i, map_obj[i] as string));
}
}
await Promise.all(download_list);
...
};We can also update the
main function as thusif (process.argv[2] === "install" || process.argv[2] === "i") {
//toml config
if (process.argv.length == 3) {
...
}else{
//craete an empty JSON map
let cmd_map: JsonMap = {};
...
await install(cmd_map);
for (let dep in cmd_map) {
obj["dependencies"][dep] = cmd_map[dep];
}
let toml_str = stringifyObj(obj);
writeFile("./pkg.toml", toml_str);
}As we can see, after successfully installing the dependency, we update the parsed
TOML string and write it back to the pkg.toml file.If you noticed in the
install function, we have a huge JsonMap that contains all the dependencies to the nth depth in a single place. We can use that to create our dependency map
const install = async (depenecyMap: JsonMap) => {
...
if (!fs.existsSync(`./pkg.lock.toml`)) {
let toml_str = stringifyObj(dependecy_graph);
writeFile("./pkg.lock.toml", toml_str);
} else {
const toml_data = await readFile("./pkg.lock.toml");
const obj = parseToml(toml_data);
let toml_str = stringifyObj({ ...dependecy_graph, ...obj });
writeFile("./pkg.lock.toml", toml_str);
}
};
And just like that, we have our dependency map for all the packages managed by our package manager. 😃
Now let’s add some extra features that would make our package manager more attractive.
Running specific commands from the TOML file akin to npm run <command>
For those familiar with how
npm or yarn works, we would know that we can run custom commands specified in the package.json file inside the scripts sectioneg
{
"name": "application",
"version": "1.0.0",
"scripts": {
"start": "ts-node index.ts"
},
"description": "a simple application",
"main": "index.js",
}To start our app, we just need to run
npm run start. Let’s try implementing that for our package manager. But instead we would use ./index.ts exec <commad>.
We should also allow users to specify the
scripts section in their TOML file as such[scripts]
build = "node build.js"
test = "node test.js"
start = "ts-node index.ts"
First we need to read input after the
exec arg from the CLIelse if (process.argv[2] === "exec") {
if (process.argv.length < 4) {
console.error("invalid exec arg supplied");
} else {
//get scripts
const script = obj["scripts"];
execScript(script[process.argv[3]]);
}
}Remember that the
obj is the parsed TOML object. We get all the JsonMap from the scripts section and pass in the 4th arg. Eg ./index.ts exec start would pass in the command ts-node index.ts into the execScript
const execScript = (command: string): void => {
//parse the commad
let cmd = command.split(" ");
const child = spawn(cmd[0], cmd.slice(1));
let ScriptOutput = "";
child.stdout.setEncoding("utf8");
child.stdout.on("data", (data) => {
console.log(data);
data = data.toString();
ScriptOutput += data;
});
child.on("close", (code) => {
if (code !== 0) {
console.log(`exited with status ${code}`);
}
});
};An voila we can execute our custom scripts 🙃
Deleting package(s)
Now let’s add an option to delete packages after they are installed. For this to happen, reading all the package list from the
node_modules seems like an expensive operation. So I would rather use the pkg.lock.toml as the ultimate source of truth.
We need to consider something when developing the delete module. A package being deleted might be required by another package to operate. eg we might want to delete
express, but another library requires express to operate. So if we just delete express, all other libraries that depend on it would throw an error when used.One way we could tackle this is simple; if a dependency is deleted, we check the dependency map on our
pkg.lock.toml file. If other dependencies rely on it, I simply remove them from the pkg.lock.toml and pkg.toml file else I delete the folder and all it’s occurrences in the pkg.lock.toml and pkg.toml.else if (process.argv[2] === "delete" || process.argv[2] === "d") {
if (process.argv.length < 4) {
console.error("invalid delete arg supplied");
} else {
//get scripts
const toml_lock_data = await readFile("./pkg.lock.toml");
const obj = parseToml(toml_lock_data);
deleteDep(obj, process.argv.slice(3));
}
}const deleteDep = async (payload: JsonMap, delete_item: string[]) => {
for (let item of delete_item) {
let dep = payload[item] as JsonMap;
delete payload[item];
if (dep) {
let check = false;
//check if a dependecy sub dependecy relies on this
for (let sub_dep in payload) {
if (payload[sub_dep][item]) {
check = true;
}
}
if (check === false) {
removeSync(`./${dir_name}/${item}`);
console.log(`deleting ${item} from depency list`);
}
//update config file
const toml_data = await readFile("./pkg.toml");
const obj = parseToml(toml_data);
if (obj["dependencies"][item]) {
delete obj["dependencies"][item];
} else {
delete obj["devDependncies"][item];
}
let toml_config_str = stringifyObj(obj);
writeFile("./pkg.toml", toml_config_str);
//update the lock file
let toml_str = stringifyObj(payload);
writeFile("./pkg.lock.toml", toml_str);
} else {
console.error(`parent dependecy ${item} not found`);
}
}
};
And just like that, we have a semi-functional package manager.
This exercise has opened my eyes to some of the inner workings of JavaScript package managers, and trying to mimic them was definitely a worthwhile experience. Thanks for reading.
