Since telling is nothing without the showing. We have a very simple implementation of the above work.
The resource being created is a text document with time set every ten second apart for an iteration of ten.
Now this is a trivial application but should be able to show what we need. You can get a copy of the source files from github https://github.com/prodeveloper/long_running_script
Without further a do here are the files.
For this particular implementation I have chosen to choose one file for progress and one for storage, they are imaginatively named store.txt and progress.txt. For the application to work this files must be readable and writable to apache.
For your own application you definitely want to use a better database not to mention provide a way of identifying the different workers.
Next time we will talk about libraries and tools that can help you in your development process.
Remember to follow me on twitter/ subscribe to my feed.
Till next time happy coding!!
by Jacob Chencha
The design mock up
What we have above is the proposed app overview.
Here is a brief overview of what is actually happening:
- Requester (User) Sends a request for a resource to the system. The resource could be HTML, Report etc
- Controller receives the requests and returns a HTTP 202 response. See http://en.wikipedia.org/wiki/List_of_HTTP_status_codes for more details on HTTP status codes.
- Controller sends job parameters to model
- Controllers sends the url that will call the appropriate worker to the queue manager
- Queue manager receives job request
- Queue manager evaluates systems resources and decides sends requests to workers. This can be done via curl. The queue manager should not wait for a response
- Worker does the time intensive task and on completion (successfully or not) sends output back to the Queue manager
- The Queue manager updates the progress to the model
- Requester(User) sends request for the current progress.
- Controller checks progress from the model, if complete returns the resource if not, returns the progress information that is available
Obviously not all work flows will fit this suggested work flow, regardless this should at least come close.
Next time we are going to go through a sample implementation of this design.
If you have any questions please drop a comment below. Remember to follow me on twitter.
by Jacob Chencha
This is the first in a five part session that will teach you the basics of designing a projects that is resource intensive.
So you now have a couple of applications under your belt, maybe even mastered a framework or two. Barring special circumstances you have come to expect response from the server in the order of milliseconds. Then baaaam comes a project with various resource intensive jobs and everything goes whack!!
Below are some of the problems that you are likely to encounter if you don’t plan for this resource hungry tasks:
1. Server will cut you off: The default limit for php scripts is 30 seconds. You could always modify this by changing the execution time http://php.net/manual/en/function.set-time-limit.php and in fact for worker scripts this maybe inevitable but doing this is really just sweeping the problem under the carpet, which is just not a good idea, there are spiders there! Furthermore if you are in a shared scripting environment then things are a tad worse for you because you just don’t have access to php.ini file and are more likely than not unable to change global values.
2. You will run out of resources: Unlike most other scripting languages, php does not repeat file descriptors. For small to medium tasks this is hardly ever a problem for bigger projects however this can escalate very fast. See http://gnuvince.wordpress.com/2008/10/28/php-wrong-for-long-running-processes-wrong-for-america/
3. Google will hate you: While Google’s ranking algorithm remains propriety, it is an open secret that page load does affect rank. Now unless you have a couple of millions to spare in advertising budget, you don’t want to find yourself in the netherworld that is page 2 of Google search!. For interested readers, you may read more here http://www.seochat.com/c/a/google-optimization-help/average-page-load-time-of-top-ranking-websites-in-google/
4. The user will hate you: LOVE YOUR USERS. We can never over emphasize this. Slow and hanging apps amount to right out insulting them. Now you may argue that this is an internal app or that you, the developer, will be the only user but that is irrelevant and besides the point, when developing think of the user as a separate entity. You will thank yourself later.
There is a lot of argument as regards php and task intensive tasks http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script I think the argument there is irrelevant besides we love PHP through the good and the bad right 🙂
Now that I have presented the challenge let me present the suggested solution. Note what I will be talking about here is design of your application and how to make it ready for long running tasks.
Watch out for part 2 of this series coming to you tomorrow.
If you like this, please follow me on twitter use the tabs to your right.
by Jacob Chencha