` components ensure that each section ends up being a link for easy `index.html#navigation` navigation and bookmarking.
+- bundling uses [Webpack](https://webpack.github.io), with [babel](http://babeljs.io/) for converting JSX and ES6+ code to plain old ES5
+- a number of custom webpack loaders are used to turn "my code" into "actually spec-compliant JSX" (insofar as there is a spec, of course):
+ - a [`p-loader`](https://github.com/Pomax/BezierInfo-2/blob/4c3be7105161d69774e622432654a5dbc62bae96/lib/p-loader.js) which replaces `{` and `}` inside paragraphs with JSX-safified versions instead, because to JSX those are templating instructions.
+ - a [`pre-loader`](https://github.com/Pomax/BezierInfo-2/blob/4c3be7105161d69774e622432654a5dbc62bae96/lib/pre-loader.js) which replaces "html sensitive" characters in `` blocks with safe equivalents, so that I can write `i < 0` rather than having to bother with remembering to use `i < 0`. Which is good: having to remember HTML entities is dumb.
+ - a [`latex-loader`](https://github.com/Pomax/BezierInfo-2/blob/4c3be7105161d69774e622432654a5dbc62bae96/lib/latex-loader.js) which does some "genuinely cool shit™"
+- There is a `` component that ensure that each section ends up providing its own fragment identifier for easy `index.html#somesection` navigation and bookmarking.
-It gets even better because for the longest time the LaTeX would simply be parsed client-side using MathJax but that made the article load really slowly, necessitating progressive content loading and all kinds of other tricks that solved symptoms, not problems: the problem that LaTeX should be presented as normal maths.
+Particularly the LaTeX loader is a thing I really like: [I wrote about this before](http://pomax.github.io/1451617530567/react-with-latex-without-needing-client-side-mathjax), but for the longest time the on-page LaTeX would simply be parsed client-side using [MathJax](https://mathjax.org) but that made the article load really slowly because I use a lot of maths. Like... **a lot**. People had to basically wait for 150+ blocks of LaTeX to be typeset by their browser, slowing down the page load, making the page unresponsive, and basically just being super annoying. Worth it for what you got in return, but still annoying.
-So instead, the latex-loader was made to extract LaTeX code, run it offline through MathJax at build time, save the result as SVG file, and then include the SVGs in the article itself. Yes, it's over 150 SVG files but the speed boost from not needing to do LaTeX parsing of 150+ blocks is tremendous.
+So, eventually, I wrote the latex-loader whose specialized role was to ingest a JSX file, extract all LaTeX code blocks, convert them *offline during the build* by running them through MathJax (using [mathjax-node](https://www.npmjs.com/package/mathjax-node)), saving the resulting SVG to file, and then generating a *new* JSX file in which each LaTeX block got replaced with `
` to effect two things: first, excellent load experience for users because their browser wasn't locking up anymore, and second, the page content wouldn't move around anymore because while client-side MathJax means there was no way to tell how much vertical space a conversion result would take up, having actual SVG files means you can just quickly check what their size is going to be and bind an image `width` and `height` value so that the DOM already knows how much space it needs even before an image is loaded.
-So all in all it's pretty sweet!
+All in all, it was *pretty* sweet!
-## And then contributors show up
+## And then contributors show up.
-I've pretty much been working on the Primer on my own since 2011, with some help from people in terms of advice or suggestions, but never in the sense that they took the code, changes something radically, and then filed a PR and discussed the changes to get them landed.
+I've been working on the Primer pretty much on my own since 2011, with some help from people in terms of advice or suggestions, but never in the sense that they took the code, changes something radically, and then filed a PR and discussed the changes to get them landed.
-In February of 2017, that changes. In the span of a week two people contacted me because they wanted to translate the article to their own language. Specifically: Chinese and Japanese. And this is where, if you're an English content creator, things get interesting because that's not just translation: that's localization. While of course translation is involved, things like how maths statement are organized also changes.
+In February of 2017, that changed. In the span of a week two people contacted me because they wanted to translate the article to their own language. Specifically: Chinese and Japanese. And this is where, if you're an English content creator, things get interesting because that's not just translation: that's localization. While of course translation is involved, things like "how maths statement are organized" also changes, and probably some other things you hadn't even thought of.
-I really wanted to take these folks up on their offer: getting the Primer translated so that the content would be more meaningful for Chinese and Japanese audiences is huge! But my code base was written by one person, for one person, for one language, and so I did the only thing I could think of:
+Naturally, I really wanted to take these folks up on their offer: getting the Primer translated so that the content is more meaningful for Chinese and Japanese audiences is huge! But my code base was written by one person, for one person, for one language, without any affordances to the possibility of eventual translation. so I did the only thing I could think of:
**I told them I was super interested in having them help out, but would need a little time to make sure they could work on the code**.
-This, I think, is the only way to respond when the world shows up at your doorstep asking to help you: you make it some tea, express your deepest gratitude and then you work your ass off to make sure it doesn't leave disappointed.
+This is where you lose contributors, by the way, but it's still the correct way to deal with people showing up wanting to help doing a thing you had never spent any time thinking of at all: when the world shows up at your doorstep asking to help you, you make it some tea, express your deepest gratitude and then while they're hopwfully distracted by the cookies you work your ass off to make sure it doesn't go "well I should go again". You make it want to stay.
-## Code base changes: let's talk about failures
+## Code base changes: let's talk about failures.
-I've worked with localized code bases before and there are some nice solutions out there for websites and apps. I figured I'd see if Transifex and Mozilla's Pontoon could be used, because that's what I've used in the past, but this is where traditional localization solutions break down.
+I've worked with localized code bases before and there are some nice solutions out there for websites and apps. I figured I'd see if [Transifex](https://www.transifex.com) and [Mozilla's Pontoon](https://pontoon.mozilla.org) could be used, because that's what I've used in the past, but this is where traditional localization solutions break down.
-Transifex is a translation service that lets you define your project in terms of key-value mappings, where you use the keys in your own content, and then you do a replacement for the values based on the locale data that you get from the transifex servers. This works really well for web apps, and general UI, but things get tricky for content like articles. In articles, where the content is structured in paragraphs and the ordering matters for the tone of the text, asking localizers to translate paragraphs or even single sentences fully detached from what comes before or after is almost guaranteed to give weird translations.
+Transifex is a translation service that lets you define your project in terms of key-value mappings, where you use the keys in your own content, and translators can create translations of the associated values to whatever language you want to have supported. You then download your latest translations from transifex and apply the key/value mappings to your content to achieve a translated web page, app, UI, etc. Now, while this works quite really well for web apps, and really well for general UI, things get tricky for content like articles. In articles, where the content is structured in paragraphs and the ordering matters for the tone of the text, asking localizers to translate paragraphs or even single sentences fully detached from what comes before or after is almost guaranteed to give weird translations.
-To make things easier in that respect, Mozilla's Pontoon project is an "on-page translation" localization system, where you load a .js file that turns all your on-page content into "double click and translate, and pontoon saves that translation for you", and while that sounds really nice, it turns out that setting it up for "not mozilla sites" is a little bit of work (and you want to, because your translations are not for a mozilla project, so you need to run your own copy of pontoon on something like a heroku app). However, even if you get that to work, you still have the problem that transifex has too: your localizers might have an easier time, but as an author you're still stuck being unable to write text yourself without then having to convert it to a weird, unreadable "mess" of `getText('section1-paragraph1')`, so you have no idea what you wrote.
+To alleviate that problem, Mozilla's Pontoon project offers a sort of "on-page translation" localization system, where you load the website and pontoon UI at the same time, and translators can "double click and translate" text on the page. While that sounds really nice, it turns out that setting it up for "not mozilla sites" is a little bit tricky: you need to run your own copy of pontoon, and a lot of the code assumes it's running on Mozilla infrastructure. However, once you do get that it work, you still have the problem that transifex has: your localizers might have an easier time, but as an author you're still stuck being unable to write text yourself without then having to convert it to an unreadable "mess" of `getText('section1-paragraph1')` in your code, so you have no idea what you wrote. You can try to get abstract keys from values (say, by hashing) but then changing even one letter in a paragraph invalidates every translation, which is typically *not* what you want. A little desync is fine for webcontent. Corrections that trickle through are infinitely preferable to "keeping either all sites on the old content, or breaking all translations for a hopefully short while as translators catch up with changes".
-Worse, and all key/value localization systems suffer from this: changing the text in the authoritative locale (in this case, en-GB) invalidates all localizations of that text. What do you do?
+So what do you do?
-The problems that come with these kinds of localization systems well too much. Either it didn't work for the localizers, or it didn't work for me as author, or it didn't work for the nature of the content, or any combination of those. So the traditional solutions for adding localization to a site were out. Not because they're unsuitable as localization solutions, but because for this project, they introduced more problems than they solved.
+The problems with the methodology of these translation/localization systems outweight the benefits they brought to my project, as they either didn't work for me as author, or didn't work for the nature of the content, so the traditional solutions for adding localization to a site were out. Not because they're unsuitable as localization solutions, but because for this project, they introduced more problems than they solved.
-## Code base changes that work: Markdown
+## Code base changes that work: Markdown.
And so after about a week of trying to make localization "easy" with string based localization services, instead I figured I'd roll my own solution because I was keeping people waiting for a week at this point and they had every reason to just go "well this Pomax guy's clearly not interested in getting help, time to look for something else to help out on".
-This is, by the way, your single biggest challenge in Open Source: if you thought it was hard to find contributors, it's ten times harder to make sure **you** respond in a timely fashion and make sure there is something for them to get started on. Don't have anything? Take a paid-time-off or vacation day and *make sure there are things for them to start on*. Or lose your contributor forever. If you leave a contributor hanging for more than a day, your project is not worth helping out on.
+This is, by the way, your single biggest challenge in Open Source: if you thought it was hard to find contributors, it's ten times harder to make sure **you** respond in a timely fashion and make sure there is something for them to get started on. Don't have anything? Take a paid-time-off or vacation day and *make sure there are things for them to start on*. Or lose your contributor. If you leave a contributor hanging for more than a day, your project isn't really worth helping out on.
-So I had to make this work, and the solution to the problem was super obvious in hindsight: just detach the content from the code. This is programming, we can make integration work in like, at least seven different ways, but the base case should be "here is the section as natural, flowing text, with embedded LaTeX where necessary and code blocks to illustrate how to program stuff. Just translate that" and so that's what I set out to achieve.
+So I had to make this work, and the solution to the problem was one that basically comes to us from the traditional translation world: detach the content from the code, and translate the content as full length pieces. This is programming, we can make "integrating disparate pieces of code and content" work in at least seventeen different ways, so it's entirely possible to offer authors and translators alike a base case where "here is the section as natural, flowing text, with embedded LaTeX where necessary and code blocks to illustrate how to program stuff. Just translate that". And so that's what I set out to achieve.
- I decided on a content format: sections would be an `index.js` for the JSX code, and a `content.en-GB.md` for my own English content.
- content would be pulled back into the JSX by... wait...
-How do you pull markdown content into a JSX file? Unlike `js` or `jsx` or `json`, markdown content can't just be imported. And this is where things go a little different than you would expect: instead of solving this problem so that I had a working "full circle", solving this problem first would prevent translations from happening. Consequenlty **I stepped away from the problem and went back to my contributors**: I had a solution *for them*, and that came first.
+How do you pull markdown content into a JSX file? Unlike `js` or `jsx` or `json`, markdown content can't just be imported in JavaScript or even in Node.js... It's a problem, but really solving this problem before getting back to my contributors would prevent translations from happening, and so *I stepped away from the problem and told them I had something they could work with now*. I had a solution *for them*, and that came first.
-Yes, it's nice to solve these things as they pop up, but the most important part is still to make sure contributors can start doing what they wanted to help you with and once I figured out how to at least split up the JSX into "JSX for the skeleton" and "Markdown for the content", that was it: I had a solution that unblocked my contributors so that *they* could now at least get started translating and making progress, even if the system for rebuilding the content wasn't done yet and I wouldn't be able to immediately deploy whatever PR they were going to throw my way.
+Yes, it's nice to solve these things as they pop up, but the most important part is still to make sure contributors can start doing what they wanted to do to help, and once I figured out how to at least split up the JSX into "JSX for the skeleton" and "Markdown for the content", I had a solution that unblocked my contributors so that *they* could now at least get started translating and making progress, even if the system for rebuilding the content wasn't done yet and I wouldn't be able to immediately deploy whatever PR they were going to throw my way.
-And so that's what happened: their needs came first. Once I figured out how to at least split the content as JSX and Markdown, I split up the article preface and first three sections as `content.en-GB.md` and told the hopefully-still-willing-to-be-contributors that if they were still interested in helping out they could now start on these files. All they had to do was copy it to `content.zh-CN.md` or `content.ja-JP.md` and then modify that as best they knew how to.
+Their needs came first.
+
+Once I figured out how to at least split up the content as JSX and Markdown, I sat down and split up the preface and first three sections for the primer, creating a few `content.en-GB.md` that set the tone for what the markdown would look like, and then I told the hopefully-still-willing-to-be-contributors that if they were still interested in helping out they could now start on these files. All they had to do was copy it to `content.zh-CN.md` or `content.ja-JP.md` and then modify that as best they knew how to.
And while they were doing that, I'd have some time to implement getting the Markdown loaded back into the JSX files to generate a site that, for visitors, was identical to the monolithic English one.
-The take-away here is primarily: **unblock your contributors before you unblock yourself**
+The take-away here is mostly that perfect solutions aren't necessary: if you're balancing priorities **unblock your contributors before you unblock yourself**
## Reintegrating sections based on JSX and Markdown
While the contributors were working on their translations, I got back to work integrating Markdown into the JSX, and after a bit of thinking the solution to how to achieve that integration was remarkably simple: *you don't*.
-I know, that sounds a bit silly, but it's not silly as you might first imagine; I solved this problem using the classic problem solving approach of "if X is hard, which Y is easy, and how do you turn X into Y". This is a general life skill when it comes to problem solving and I honestly don't practice it enough, but I practiced it here:
+I know, that sounds a bit silly, but it's not silly as you might initially think; I solved this problem using the classic problem solving approach of "if X is hard,find out which Y is easy, and find out how you turn X into Y". This is a general life skill when it comes to problem solving and I honestly don't practice it enough, but I practiced it here:
- Pulling markdown into JSX is hard,
- pulling JSX into JSX is trivial,
-- how do I convert markdown into JSX?
+- I should convert markdown strings into JSX strings.
-Well, I'm good at programming, and both markdown and JSX are, at their code, just string data in a file. And converting string data into other string data is a pretty easy thing if you know how to program. So I wrote a script called `make-locales.js` which runs through the `./components/sections` directories looking for `content.*.md` files, filters the list of locales it finds that way, turning it into a list of unique locales, and then for each locale in that list does something like:
+I'm pretty good at programming, and both markdown and JSX are, at their code, just string data in a file. As converting string data into other string data is a pretty easy thing if you know how to program, I wrote a script called `make-locales.js` which runs through the `./components/sections` directories looking for `content.*.md` files, filters the list of locales it finds that way, turning it into a list of unique locales, and then for each locale in that list does something like:
```
for (locale in locales) {
@@ -95,7 +102,7 @@ for (locale in locales) {
}
```
-Running this script builds a `content.js` file that takes a form that matches the one necessary for any Node script (which JSX files are in my codebase) to trivially import with a single `require('content')` statement. By further making sure the data inside `content.js` is keyed in the same way as the original code base organised sections, I basically had a markdown-to-JSX conversion that the original code base didn't even notice was different. Everything basically worked the same as far as it was concerned.
+Running this script builds a `content.js` file that takes a form that matches the one necessary for any Node script (which JSX files are in my code base) to trivially import with a single `require('content')` statement. By further making sure the data inside `content.js` is keyed in the same way as the original code base organised sections, I basically had a markdown-to-JSX conversion that the original code base didn't even notice was different. Everything basically worked the same as far as it was concerned.
### Further challenges: I'm not using *true* markdown
@@ -248,11 +255,11 @@ Here's the thing about localisation: you need to update *all* the content to wor
Wouldn't it be nice if in a Japanese-localiazed Primer, this:
-
+
looked like this instead?
-
+
Of course the answer is "yes, that would be lovely actually" but unfortunately this ran into a wall: MathJax, for all the love I have for it, is English. It does not understand, and in fact breaks down incredibly, when you try to feed it anything other than English. So I had two options
@@ -300,7 +307,7 @@ So directories it is: it's not clean from a dir structure perspective, but ultim
Of course, you also need to be able to *switch* locales on the site, so I wrote a small component that offers users a compellingly simple choice:
> Read this in your own language: | **English** | **日本語** | **中文** |
-> *Don't see your language listed? [Help translate this content!](https://github.com/Pomax/BezierInfo-2/wiki/localize)*
+> *Don't see your language listed? [Help translate this content!](https://github.com/Pomax/BézierInfo-2/wiki/localize)*
A simple list of languages that people can read if it's theirs, and a call-out to anyone who *wants* localized content but can't find it: this is an Open Source project, come help out!
@@ -310,7 +317,7 @@ So now the code base can be localized! Hurray!
Last question: can people read up on how to do that without needing to ask questions, and just *do it*? If not, you're not done. Yes, it's great to have everything in place but unless you also write the documentation that explains how to do what people want to do, they're going to need help, and you probably won't have time to spend on helping them, so: write the docs that take you out of the equation.
-I ended up documenting [the steps necessary to do localization](https://github.com/Pomax/BezierInfo-2/wiki/localize) on the github repo wiki, with links to the docs from the README.md, so that anyone visiting the repo --linked prominently on the Primer itself-- will immediately be able to find out what is involved and how they can make that work for them.
+I ended up documenting [the steps necessary to do localization](https://github.com/Pomax/BézierInfo-2/wiki/localize) on the github repo wiki, with links to the docs from the README.md, so that anyone visiting the repo --linked prominently on the Primer itself-- will immediately be able to find out what is involved and how they can make that work for them.
## So what's left?
diff --git a/package.json b/package.json
index 3a741e45..d4b8e4e5 100644
--- a/package.json
+++ b/package.json
@@ -4,7 +4,9 @@
"description": "pomax.github.io/bezierinfo",
"scripts": {
"localize": "node make-locales",
+ "prebuild": "node buildmark",
"build": "run-s localize less build:**",
+ "postbuild": "node buildmark",
"build:en-GB": "webpack -p",
"build:zh-CN": "cross-env LOCALE=zh-CN webpack -p",
"build:ja-JP": "cross-env LOCALE=ja-JP webpack -p",